# Website Content Crawler Pro — Markdown & RAG Chunks for AI (`hipersoft/web-content-crawler-pro`) Actor

Crawl any website to clean Markdown, text or HTML for AI, LLMs and RAG. Heading-aware chunks, sitemap and llms.txt discovery, URL filters, duplicate removal, PDF link capture, rich metadata and incremental crawls. Fast HTTP or JavaScript rendering. Pay only for pages delivered.

- **URL**: https://apify.com/hipersoft/web-content-crawler-pro.md
- **Developed by:** [Hipersoft](https://apify.com/hipersoft) (community)
- **Categories:** Developer tools, AI
- **Features:** Limited permissions
- **Stats:** 15 total users, 5 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.002 / page crawled

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website Content Crawler Pro — Clean Markdown & RAG Chunks for AI and LLMs

**Website Content Crawler Pro** crawls any website and turns it into clean **Markdown**, **plain text** or **HTML**, with **metadata**, **document links** and optional **heading-aware RAG chunks** on every page. Point it at a documentation site, blog or knowledge base and export the result as JSON, CSV, Excel or XML, ready for vector databases, LLM applications and AI agents. You pay only for pages actually delivered.

![Website Content Crawler Pro input form: start URLs, crawler type, page limits and crawl depth](https://api.apify.com/v2/key-value-stores/SUfvnaFLd9z9eBCtV/records/web-content-crawler-pro-input.png?v=6)

### What does Website Content Crawler Pro do?

This website crawler starts from your URLs, follows links within the limits you set, removes navigation, footers, sidebars and cookie banners, and saves the main content of each page as one dataset row. It can also discover pages from **sitemaps** and **llms.txt**, skip **duplicate pages**, re-crawl only what **changed since a date**, and split each page into **chunks** for embeddings. Use the fast HTTP mode for most sites, or the browser mode for JavaScript-rendered sites and single-page apps.

### What data can you scrape from a website?

| Data | Fields |
|------|--------|
| 📝 Page content | `markdown`, `text`, `html`, `htmlUrl` |
| 🧩 RAG chunks | `chunks` (id, index, text, charCount, tokenEstimate, headings) |
| 🏷️ Page details | `url`, `title`, `description`, `canonical`, `lang`, `author`, `siteName` |
| 📊 Size and change tracking | `wordCount`, `tokenEstimate`, `contentHash` |
| 🔗 Links | `links`, `fileLinks` (PDF, Word, Excel, PowerPoint, CSV, EPUB, ZIP) |
| 🧠 Metadata | `metadata`: keywords, publish and update dates, image, robots, Open Graph tags, JSON-LD, key response headers |
| 🕷️ Crawl details | `crawl`: requested and loaded URL, time, referrer, depth, HTTP status, how the page was found |

### Use cases

- **RAG and AI knowledge bases**: turn documentation into chunks with heading context, ready to embed
- **Custom GPTs and AI agents**: give an assistant a clean copy of your docs, help center or website
- **LLM datasets**: collect clean text from many pages without site chrome
- **Keeping a vector database fresh**: re-crawl only recently changed pages, and use `contentHash` and chunk ids to upsert
- **Document discovery**: list every PDF and Office file a website links to
- **Content migration**: export a blog or docs site to Markdown
- **Content and SEO research**: pull titles, descriptions, word counts and metadata across a domain

### How to crawl a website to Markdown

1. Open the actor and click **Try for free**.
2. Paste one or more **start URLs**.
3. Turn on **Follow links**, and set **Max pages**, **Max depth** and any URL filters.
4. Optionally set a **RAG chunk size**, and enable **sitemaps** or **llms.txt** discovery.
5. Click **Start**, then export the results as JSON, CSV or Excel, or fetch them through the API.

### Input

Example input: crawl one section of a documentation site into 2,000-character chunks.

```json
{
  "urls": [
    "https://docs.python.org/3/tutorial/"
  ],
  "crawl": true,
  "restrictToStartPath": true,
  "maxCrawlPages": 100,
  "maxCrawlDepth": 3,
  "chunkSize": 2000,
  "chunkOverlap": 200
}
```

| Field | Description |
|-------|-------------|
| `urls` / `startUrls` | Where to start. Bare domains such as `example.com` work too. |
| `crawlerType` | `http` (fast, the default) or `browser` (renders JavaScript). |
| `crawl` | Follow links from the start URLs. Off = only the pages you list. |
| `maxCrawlPages` | Most pages to request, including duplicates and failures. |
| `maxResults` | Stop after this many pages are stored (0 = no limit). |
| `maxCrawlDepth` | Link hops from a start URL. |
| `sameDomainOnly` | Stay on the start URL's domain. |
| `restrictToStartPath` | Stay under the start URL's path, e.g. only `/docs/`. |
| `includeUrlGlobs` / `excludeUrlGlobs` | Glob filters for followed links, sitemap and llms.txt pages. |
| `useSitemaps` | Discover pages from the site's sitemaps. |
| `useLlmsTxt` | Crawl the site's llms.txt and the pages it lists. |
| `respectRobotsTxtFile` | Skip pages disallowed by robots.txt. |
| `onlyCrawlPagesModifiedSince` | Incremental crawl: a date such as `2026-01-31`. |
| `saveMarkdown` / `saveText` / `saveHtml` / `saveHtmlAsFile` | Output formats. |
| `saveLinks` / `saveFileLinks` / `saveMetadata` | Extra fields on each row. |
| `chunkSize` / `chunkOverlap` | RAG chunks in characters (about 4 characters = 1 token). |
| `textMaxChars` | Cut each page's text and Markdown to this length. |
| `htmlTransformer` | `readability` (main content), `cleanBody` (full page without site chrome) or `none` (full body). |
| `removeBoilerplate` | Remove navigation, headers, footers, sidebars and pop-ups. |
| `removeElementsCssSelector` / `keepElementsCssSelector` | Extra elements to remove, or the only elements to keep. |
| `removeCookieWarnings` | Remove cookie and consent banners. |
| `ignoreCanonicalUrl` | Keep pages that would otherwise be dropped as duplicates. |
| `dynamicContentWaitSecs` / `waitForSelector` | Browser mode: wait for dynamic content. |
| `maxConcurrency` / `maxRequestRetries` / `requestTimeoutSecs` | Performance and retries. |
| `proxyConfiguration` | Proxy settings. Leave the default. |

### Output

![Website Content Crawler Pro output example: one crawled page with Markdown, metadata, chunks and document links as JSON](https://api.apify.com/v2/key-value-stores/SUfvnaFLd9z9eBCtV/records/web-content-crawler-pro-output.png?v=6)

One row per page. This is a real row, with long text shortened here for reading:

```json
{
  "url": "https://docs.python.org/3/tutorial/controlflow.html",
  "title": "4. More Control Flow Tools — Python 3.15.2 documentation",
  "description": "As well as the while statement just introduced, Python uses a few more that we will encounter in this chapter. if Statements: Perhaps the most well-known statement type is the if statement. For exa...",
  "canonical": "https://docs.python.org/3/tutorial/controlflow.html",
  "lang": "en",
  "author": null,
  "siteName": "Python documentation",
  "wordCount": 5628,
  "markdown": "# 4. More Control Flow Tools\n\nAs well as the [`while`](https://docs.python.org/3/reference/compound_stmts.html#while) statement just introduced, Python uses a few more that we will encounter in this chapter.\n\n## 4.1. `if` Statements\n\nPerhap …",
  "text": "As well as the while statement just introduced, Python uses a few more\nthat we will encounter in this chapter.\n\n4.1. if Statements\n\nPerhaps the most well-known statement type is the if statement. For\nexample:\n\n>>> x = int(input(\"Please ente …",
  "fileLinks": [],
  "contentHash": "5c6d588303fbf85e0672206f4af07fa4dc3bff371eb0353e5d9e794b2ae05bed",
  "tokenEstimate": 10176,
  "crawl": {
    "requestedUrl": "https://docs.python.org/3/tutorial/controlflow.html",
    "loadedUrl": "https://docs.python.org/3/tutorial/controlflow.html",
    "loadedTime": "2026-11-04T10:14:08.931Z",
    "referrerUrl": "https://docs.python.org/3/tutorial/",
    "depth": 1,
    "httpStatusCode": 200,
    "source": "link"
  },
  "metadata": {
    "canonicalUrl": "https://docs.python.org/3/tutorial/controlflow.html",
    "title": "4. More Control Flow Tools — Python 3.15.2 documentation",
    "description": "As well as the while statement just introduced, Python uses a few more that we will encounter in this chapter. if Statements: Perhaps the most well-known statement type is the if statement. For exa...",
    "author": null,
    "keywords": null,
    "languageCode": "en",
    "publishedAt": null,
    "modifiedAt": null,
    "image": "https://docs.python.org/3.15/_images/social_previews/summary_tutorial_controlflow_a44c5fc7.png",
    "robots": null,
    "openGraph": [
      {
        "property": "og:title",
        "content": "4. More Control Flow Tools"
      },
      {
        "property": "og:type",
        "content": "website"
      }
    ],
    "jsonLd": [],
    "headers": {
      "content-type": "text/html",
      "last-modified": "Wed, 04 Nov 2026 09:14:47 GMT",
      "etag": "\"6aeaf787-22392\""
    }
  }
}
```

#### RAG chunks

With `chunkSize` set, each row also has a `chunks` array. Real example from the same run:

````json
{
  "chunks": [
    {
      "id": "081d1e1b084bddc2",
      "index": 2,
      "text": "== 'inactive':\n        del users[user]\n\n# Strategy:  Create a new collection\nactive_users = {}\nfor user, status in users.items():\n    if status == 'active':\n        active_users[user] = status\n```\n## 4.3. The [`range()`](https://docs.python.org/3/builtins/stdt …",
      "charCount": 1707,
      "tokenEstimate": 427,
      "headings": [
        "4. More Control Flow Tools",
        "4.3. The `range()` Function"
      ]
    }
  ]
}
````

Need one row per chunk for your vector database? Add `unwind=chunks` to the dataset items API URL.

#### Document links

`fileLinks` lists the documents a page links to. Real example:

```json
{
  "url": "https://www.gov.uk/guidance/section-62a-planning-application-s62a20250091-265-267-church-road-bristol-bs5-9hu",
  "fileLinks": [
    {
      "url": "https://assets.publishing.service.gov.uk/media/685977e66e0a19f50feca7b8/S62A_2025_0091_Decision_Notice_and_Statement_of_Reasons.pdf",
      "type": "pdf",
      "text": "Decision Notice and Statement of Reasons"
    },
    {
      "url": "https://assets.publishing.service.gov.uk/media/67d40d811b26cbdf9b851d8f/250303_cover_letter_PR002661_FINAL_checked.pdf",
      "type": "pdf",
      "text": "250303 cover letter PR002661 FINAL"
    }
  ]
}
```

#### Output schema

| Field | Type | Description |
|-------|------|-------------|
| `url` | string (URL) | Page URL after redirects. |
| `title` | string | Page title. |
| `description` | string | Meta description. |
| `canonical` | string (URL) | Canonical URL declared by the page. |
| `lang` | string | Page language code. |
| `author` | string | Author, when the page declares one. |
| `siteName` | string | Site name. |
| `wordCount` | integer | Words in the extracted content. |
| `tokenEstimate` | integer | Rough token count of the Markdown (characters ÷ 4). |
| `contentHash` | string | SHA-256 of the text. It changes when the content changes. |
| `markdown` | string | Main content as Markdown, with tables and code blocks. |
| `text` | string | Main content as plain text. |
| `html` | string | Cleaned content HTML (`saveHtml`). |
| `htmlUrl` | string (URL) | Link to the stored HTML file (`saveHtmlAsFile`). |
| `links` | array | All links on the page (`saveLinks`). |
| `fileLinks` | array | Document links: `url`, `type`, `text`. |
| `chunks` | array | RAG chunks: `id`, `index`, `text`, `charCount`, `tokenEstimate`, `headings`. |
| `crawl` | object | `requestedUrl`, `loadedUrl`, `loadedTime`, `referrerUrl`, `depth`, `httpStatusCode`, `source`. |
| `metadata` | object | `canonicalUrl`, `title`, `description`, `author`, `keywords`, `languageCode`, `publishedAt`, `modifiedAt`, `image`, `robots`, `openGraph`, `jsonLd`, `headers`. |
| `truncated` | boolean | Present when a very large page was shortened to fit the dataset's row limit. |

The run also saves a `CRAWL_REPORT` record in its key-value store, listing URLs that were not delivered and why, plus how many pages were skipped as duplicates.

### Need more web data?

- [Sitemap Extractor](https://apify.com/hipersoft/sitemap-extractor): list every URL in a website's sitemaps
- [Text Splitter & Chunker](https://apify.com/hipersoft/text-splitter-chunker): chunk text you already have
- [PDF to Text Extractor](https://apify.com/hipersoft/pdf-to-text-extractor): get the text from the PDFs listed in `fileLinks`
- [Website Change Monitor](https://apify.com/hipersoft/website-change-monitor): get alerted when a page changes

### FAQ

**How does billing work?**
Pay per event: a small fee per run start and a fee per page delivered. Pages that fail, return an error, are blocked by the site, have no readable content or are skipped as duplicates are not charged. Set a maximum cost per run and the crawl stops there, storing only what your budget covers. See the **Pricing** tab for current rates. Platform usage is listed there too.

**How do I crawl only part of a website?**
Use `restrictToStartPath` to stay under the start URL's path, or `includeUrlGlobs` and `excludeUrlGlobs` for exact patterns such as `https://example.com/docs/**`.

**Which crawler type should I use?**
Start with `http`: it is faster and costs less. If a page comes back with no content, the run tells you, and you can switch `crawlerType` to `browser` for that site. Browser mode never switches on by itself.

**What chunk size should I use for RAG?**
`chunkSize` 2000 with `chunkOverlap` 200 is a good start (roughly 500 tokens per chunk). Chunks follow the page's headings, and each one records its heading path so you can add context to embeddings.

**Can it download PDFs?**
It captures links to PDFs and other documents in `fileLinks`, with their anchor text. It does not download the files or read their contents. Pair it with a PDF actor for that.

**How do I re-crawl only what changed?**
Set `onlyCrawlPagesModifiedSince` to the date of your last crawl. Pages dated earlier are skipped and not charged. Pages with no date on the site are delivered. You can also compare `contentHash` between runs.

**Does it remove duplicates?**
Yes. Pages found while crawling are dropped when they repeat a delivered canonical URL or identical content. Start URLs are always delivered. Set `ignoreCanonicalUrl` to keep everything.

**Why did some pages fail?**
Open `CRAWL_REPORT` in the run's key-value store. It gives a reason per URL, for example `http-404`, `blocked-by-site` or `no-readable-content-without-javascript`. Sites that block automated visitors are not worked around: those pages are reported and not charged.

**How much memory should I set?**
The default suits both modes. For `http` runs you can lower it to reduce platform usage. In our tests a 300-page `http` crawl peaked under 700 MB, so 1 GB was enough and cost less. Keep the default for `browser` runs.

**What happens if a run is interrupted?**
The crawl keeps its progress, so a resumed run continues without delivering or charging for finished pages again.

**Is it legal to crawl websites?**
This actor collects publicly available pages. You are responsible for following each site's terms and the laws that apply to your data, especially for personal data. You can enable `respectRobotsTxtFile`.

**Can I connect this to other apps and services?**
The Website Content Crawler Pro can be connected with almost any cloud service or web app thanks to [integrations on the Apify platform](https://apify.com/integrations). It works with [Make](https://apify.com/integrations/make), [Zapier](https://apify.com/integrations/zapier), [Slack](https://docs.apify.com/platform/integrations/slack), [Airbyte](https://docs.apify.com/platform/integrations/airbyte), [GitHub](https://docs.apify.com/platform/integrations/github), [Google Drive](https://docs.apify.com/platform/integrations/drive) and [many more](https://apify.com/integrations), plus the [Apify API](https://docs.apify.com/api/v2), JavaScript/Python clients and MCP. Or use [webhooks](https://docs.apify.com/platform/integrations/webhooks) to trigger an action whenever a run finishes.

### Feedback

How did it go? Good or bad, a [review](https://console.apify.com/actors/NFcgd7wCQ1gh2JcS6/reviews) helps other users decide and tells us what to improve. Something broken or missing? Open an [issue](https://apify.com/hipersoft/web-content-crawler-pro/issues) and we'll look at it.

### Related Actors

- [Website Content Crawler](https://apify.com/hipersoft/website-content-crawler): the lightweight text-only crawler
- [Web to Markdown](https://apify.com/hipersoft/web-to-markdown): convert single pages to Markdown
- [Website Contact Scraper](https://apify.com/hipersoft/website-contact-scraper): emails, phones and social links from any site
- [Bulk URL Checker](https://apify.com/hipersoft/bulk-url-checker): status, redirects and metadata for a list of URLs

### Notes

Original clean-room implementation. Returns only public data; you are responsible for compliance with each website's terms. Not affiliated with any site it crawls.

# Actor input Schema

## `urls` (type: `array`):

URLs to crawl. One per line.

## `startUrls` (type: `array`):

Advanced: start URLs as request objects.

## `crawlerType` (type: `string`):

http = fast and cheap, for sites that serve their content in the page. browser = renders JavaScript for single-page apps and client-rendered sites.

## `crawl` (type: `boolean`):

Crawl internal links from the start URLs (otherwise only the given URLs, plus any sitemap or llms.txt pages you enable).

## `maxCrawlPages` (type: `integer`):

Maximum pages to request in total, including duplicates and pages that fail. Keeps a crawl from running away.

## `maxResults` (type: `integer`):

Stop once this many pages are stored in the dataset (0 = no limit). Unlike Max pages, duplicates and failed pages do not count.

## `maxCrawlDepth` (type: `integer`):

How many link-hops from a start URL to follow.

## `sameDomainOnly` (type: `boolean`):

Only follow links on the same domain as the start URL.

## `restrictToStartPath` (type: `boolean`):

Only crawl pages under the path of a start URL: https://site.com/docs/ stays in /docs/. Off = the whole domain.

## `includeUrlGlobs` (type: `array`):

Only crawl URLs matching these glob patterns, e.g. https://site.com/docs/\*\*.

## `excludeUrlGlobs` (type: `array`):

Skip URLs matching these glob patterns.

## `useSitemaps` (type: `boolean`):

Also discover pages from each site's sitemaps (including sitemap indexes and gzipped files declared in robots.txt). Your URL filters apply.

## `useLlmsTxt` (type: `boolean`):

Look for an llms.txt file at each site's root, then crawl it and the pages it lists. Your URL filters apply.

## `respectRobotsTxtFile` (type: `boolean`):

Skip pages the site's robots.txt disallows.

## `onlyCrawlPagesModifiedSince` (type: `string`):

Incremental crawl: a date such as 2026-01-31 or 2026-01-31T12:00:00Z. Pages that the sitemap, the server or the page itself date earlier are not delivered and not charged. Pages with no date are delivered.

## `saveMarkdown` (type: `boolean`):

Output clean Markdown (ideal for LLMs/RAG).

## `saveText` (type: `boolean`):

Output plain text.

## `saveHtml` (type: `boolean`):

Include the cleaned content HTML in the dataset row.

## `saveHtmlAsFile` (type: `boolean`):

Store the cleaned HTML of each page in the key-value store and add its link as htmlUrl. Keeps the dataset small.

## `saveLinks` (type: `boolean`):

Include the list of links found on each page.

## `saveFileLinks` (type: `boolean`):

List links to PDF, Word, Excel, PowerPoint, CSV, EPUB and ZIP files found on each page in fileLinks. The files themselves are not downloaded.

## `saveMetadata` (type: `boolean`):

Add the metadata object: canonical URL, keywords, publish and update dates, image, Open Graph tags, JSON-LD and key response headers.

## `chunkSize` (type: `integer`):

Split each page into heading-aware chunks of at most this many characters, in the chunks field (about 4 characters = 1 token). 0 = off. Try 2000.

## `chunkOverlap` (type: `integer`):

Characters repeated from the previous chunk, up to half the chunk size. Try 200.

## `textMaxChars` (type: `integer`):

Truncate each page's text/markdown to this many characters (0 = no limit).

## `htmlTransformer` (type: `string`):

readability = main article, with an automatic fallback when it would lose most of the page. cleanBody = the whole page without navigation, headers, footers and sidebars. none = the whole page body, untouched.

## `removeBoilerplate` (type: `boolean`):

Strip navigation, headers, footers, sidebars, breadcrumbs, share buttons and pop-ups before extraction. Not applied when Content extraction is None.

## `removeElementsCssSelector` (type: `string`):

CSS selector of extra elements to strip before extraction (e.g. .ads, #promo).

## `keepElementsCssSelector` (type: `string`):

If set, keep only content inside the elements matching this CSS selector.

## `removeCookieWarnings` (type: `boolean`):

Strip common cookie and consent banners.

## `ignoreCanonicalUrl` (type: `boolean`):

By default, pages found while crawling are dropped (and not charged) when they repeat a delivered canonical URL or identical content. Start URLs are always delivered. Enable to keep every page.

## `dynamicContentWaitSecs` (type: `integer`):

Browser mode: wait this many seconds for JS content to load.

## `waitForSelector` (type: `string`):

Browser mode: wait until this CSS selector appears.

## `maxConcurrency` (type: `integer`):

Max pages crawled in parallel.

## `maxRequestRetries` (type: `integer`):

How many times to retry a page after a network or server error. Failed pages are not charged.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for a page to load.

## `proxyConfiguration` (type: `object`):

Proxy configuration. Leave the default.

## Actor input object example

```json
{
  "urls": [
    "https://docs.apify.com/academy/web-scraping-for-beginners"
  ],
  "crawlerType": "http",
  "crawl": false,
  "maxCrawlPages": 20,
  "maxResults": 0,
  "maxCrawlDepth": 3,
  "sameDomainOnly": true,
  "restrictToStartPath": false,
  "useSitemaps": false,
  "useLlmsTxt": false,
  "respectRobotsTxtFile": false,
  "saveMarkdown": true,
  "saveText": true,
  "saveHtml": false,
  "saveHtmlAsFile": false,
  "saveLinks": false,
  "saveFileLinks": true,
  "saveMetadata": true,
  "chunkSize": 0,
  "chunkOverlap": 0,
  "textMaxChars": 0,
  "htmlTransformer": "readability",
  "removeBoilerplate": true,
  "removeCookieWarnings": true,
  "ignoreCanonicalUrl": false,
  "dynamicContentWaitSecs": 0,
  "maxConcurrency": 10,
  "maxRequestRetries": 3,
  "requestTimeoutSecs": 60,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `crawlReport` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://docs.apify.com/academy/web-scraping-for-beginners"
    ],
    "maxCrawlPages": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("hipersoft/web-content-crawler-pro").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": ["https://docs.apify.com/academy/web-scraping-for-beginners"],
    "maxCrawlPages": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("hipersoft/web-content-crawler-pro").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://docs.apify.com/academy/web-scraping-for-beginners"
  ],
  "maxCrawlPages": 20
}' |
apify call hipersoft/web-content-crawler-pro --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,hipersoft/web-content-crawler-pro"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/NFcgd7wCQ1gh2JcS6/builds/KUfggAfIS5YbkRN8w/openapi.json