Website Content Crawler Pro — Markdown & RAG Chunks for AI

Website Content Crawler Pro — Clean Markdown & RAG Chunks for AI and LLMs

Website Content Crawler Pro crawls any website and turns it into clean Markdown, plain text or HTML, with metadata, document links and optional heading-aware RAG chunks on every page. Point it at a documentation site, blog or knowledge base and export the result as JSON, CSV, Excel or XML, ready for vector databases, LLM applications and AI agents. You pay only for pages actually delivered.

What does Website Content Crawler Pro do?

This website crawler starts from your URLs, follows links within the limits you set, removes navigation, footers, sidebars and cookie banners, and saves the main content of each page as one dataset row. It can also discover pages from sitemaps and llms.txt, skip duplicate pages, re-crawl only what changed since a date, and split each page into chunks for embeddings. Use the fast HTTP mode for most sites, or the browser mode for JavaScript-rendered sites and single-page apps.

What data can you scrape from a website?

Data	Fields
📝 Page content	markdown, text, html, htmlUrl
🧩 RAG chunks	chunks (id, index, text, charCount, tokenEstimate, headings)
🏷️ Page details	url, title, description, canonical, lang, author, siteName
📊 Size and change tracking	wordCount, tokenEstimate, contentHash
🔗 Links	links, fileLinks (PDF, Word, Excel, PowerPoint, CSV, EPUB, ZIP)
🧠 Metadata	metadata: keywords, publish and update dates, image, robots, Open Graph tags, JSON-LD, key response headers
🕷️ Crawl details	crawl: requested and loaded URL, time, referrer, depth, HTTP status, how the page was found

Use cases

• RAG and AI knowledge bases: turn documentation into chunks with heading context, ready to embed
• Custom GPTs and AI agents: give an assistant a clean copy of your docs, help center or website
• LLM datasets: collect clean text from many pages without site chrome
• Keeping a vector database fresh: re-crawl only recently changed pages, and use contentHash and chunk ids to upsert
• Document discovery: list every PDF and Office file a website links to
• Content migration: export a blog or docs site to Markdown
• Content and SEO research: pull titles, descriptions, word counts and metadata across a domain

How to crawl a website to Markdown

1. Open the actor and click Try for free.
2. Paste one or more start URLs.
3. Turn on Follow links, and set Max pages, Max depth and any URL filters.
4. Optionally set a RAG chunk size, and enable sitemaps or llms.txt discovery.
5. Click Start, then export the results as JSON, CSV or Excel, or fetch them through the API.

Input

Example input: crawl one section of a documentation site into 2,000-character chunks.

{
  "urls": [
    "https://docs.python.org/3/tutorial/"
  ],
  "crawl": true,
  "restrictToStartPath": true,
  "maxCrawlPages": 100,
  "maxCrawlDepth": 3,
  "chunkSize": 2000,
  "chunkOverlap": 200
}

Field	Description
urls / startUrls	Where to start. Bare domains such as example.com work too.
crawlerType	http (fast, the default) or browser (renders JavaScript).
crawl	Follow links from the start URLs. Off = only the pages you list.
maxCrawlPages	Most pages to request, including duplicates and failures.
maxResults	Stop after this many pages are stored (0 = no limit).
maxCrawlDepth	Link hops from a start URL.
sameDomainOnly	Stay on the start URL's domain.
restrictToStartPath	Stay under the start URL's path, e.g. only /docs/.
includeUrlGlobs / excludeUrlGlobs	Glob filters for followed links, sitemap and llms.txt pages.
useSitemaps	Discover pages from the site's sitemaps.
useLlmsTxt	Crawl the site's llms.txt and the pages it lists.
respectRobotsTxtFile	Skip pages disallowed by robots.txt.
onlyCrawlPagesModifiedSince	Incremental crawl: a date such as 2026-01-31.
saveMarkdown / saveText / saveHtml / saveHtmlAsFile	Output formats.
saveLinks / saveFileLinks / saveMetadata	Extra fields on each row.
chunkSize / chunkOverlap	RAG chunks in characters (about 4 characters = 1 token).
textMaxChars	Cut each page's text and Markdown to this length.
htmlTransformer	readability (main content), cleanBody (full page without site chrome) or none (full body).
removeBoilerplate	Remove navigation, headers, footers, sidebars and pop-ups.
removeElementsCssSelector / keepElementsCssSelector	Extra elements to remove, or the only elements to keep.
removeCookieWarnings	Remove cookie and consent banners.
ignoreCanonicalUrl	Keep pages that would otherwise be dropped as duplicates.
dynamicContentWaitSecs / waitForSelector	Browser mode: wait for dynamic content.
maxConcurrency / maxRequestRetries / requestTimeoutSecs	Performance and retries.
proxyConfiguration	Proxy settings. Leave the default.

Output

One row per page. This is a real row, with long text shortened here for reading:

{
  "url": "https://docs.python.org/3/tutorial/controlflow.html",
  "title": "4. More Control Flow Tools — Python 3.15.2 documentation",
  "description": "As well as the while statement just introduced, Python uses a few more that we will encounter in this chapter. if Statements: Perhaps the most well-known statement type is the if statement. For exa...",
  "canonical": "https://docs.python.org/3/tutorial/controlflow.html",
  "lang": "en",
  "author": null,
  "siteName": "Python documentation",
  "wordCount": 5628,
  "markdown": "# 4. More Control Flow Tools\n\nAs well as the [`while`](https://docs.python.org/3/reference/compound_stmts.html#while) statement just introduced, Python uses a few more that we will encounter in this chapter.\n\n## 4.1. `if` Statements\n\nPerhap …",
  "text": "As well as the while statement just introduced, Python uses a few more\nthat we will encounter in this chapter.\n\n4.1. if Statements\n\nPerhaps the most well-known statement type is the if statement. For\nexample:\n\n>>> x = int(input(\"Please ente …",
  "fileLinks": [],
  "contentHash": "5c6d588303fbf85e0672206f4af07fa4dc3bff371eb0353e5d9e794b2ae05bed",
  "tokenEstimate": 10176,
  "crawl": {
    "requestedUrl": "https://docs.python.org/3/tutorial/controlflow.html",
    "loadedUrl": "https://docs.python.org/3/tutorial/controlflow.html",
    "loadedTime": "2026-11-04T10:14:08.931Z",
    "referrerUrl": "https://docs.python.org/3/tutorial/",
    "depth": 1,
    "httpStatusCode": 200,
    "source": "link"
  },
  "metadata": {
    "canonicalUrl": "https://docs.python.org/3/tutorial/controlflow.html",
    "title": "4. More Control Flow Tools — Python 3.15.2 documentation",
    "description": "As well as the while statement just introduced, Python uses a few more that we will encounter in this chapter. if Statements: Perhaps the most well-known statement type is the if statement. For exa...",
    "author": null,
    "keywords": null,
    "languageCode": "en",
    "publishedAt": null,
    "modifiedAt": null,
    "image": "https://docs.python.org/3.15/_images/social_previews/summary_tutorial_controlflow_a44c5fc7.png",
    "robots": null,
    "openGraph": [
      {
        "property": "og:title",
        "content": "4. More Control Flow Tools"
      },
      {
        "property": "og:type",
        "content": "website"
      }
    ],
    "jsonLd": [],
    "headers": {
      "content-type": "text/html",
      "last-modified": "Wed, 04 Nov 2026 09:14:47 GMT",
      "etag": "\"6aeaf787-22392\""
    }
  }
}

RAG chunks

With chunkSize set, each row also has a chunks array. Real example from the same run:

{
  "chunks": [
    {
      "id": "081d1e1b084bddc2",
      "index": 2,
      "text": "== 'inactive':\n        del users[user]\n\n# Strategy:  Create a new collection\nactive_users = {}\nfor user, status in users.items():\n    if status == 'active':\n        active_users[user] = status\n```\n## 4.3. The [`range()`](https://docs.python.org/3/builtins/stdt …",
      "charCount": 1707,
      "tokenEstimate": 427,
      "headings": [
        "4. More Control Flow Tools",
        "4.3. The `range()` Function"
      ]
    }
  ]
}

Need one row per chunk for your vector database? Add unwind=chunks to the dataset items API URL.

Document links

fileLinks lists the documents a page links to. Real example:

{
  "url": "https://www.gov.uk/guidance/section-62a-planning-application-s62a20250091-265-267-church-road-bristol-bs5-9hu",
  "fileLinks": [
    {
      "url": "https://assets.publishing.service.gov.uk/media/685977e66e0a19f50feca7b8/S62A_2025_0091_Decision_Notice_and_Statement_of_Reasons.pdf",
      "type": "pdf",
      "text": "Decision Notice and Statement of Reasons"
    },
    {
      "url": "https://assets.publishing.service.gov.uk/media/67d40d811b26cbdf9b851d8f/250303_cover_letter_PR002661_FINAL_checked.pdf",
      "type": "pdf",
      "text": "250303 cover letter PR002661 FINAL"
    }
  ]
}

Output schema

Field	Type	Description
url	string (URL)	Page URL after redirects.
title	string	Page title.
description	string	Meta description.
canonical	string (URL)	Canonical URL declared by the page.
lang	string	Page language code.
author	string	Author, when the page declares one.
siteName	string	Site name.
wordCount	integer	Words in the extracted content.
tokenEstimate	integer	Rough token count of the Markdown (characters ÷ 4).
contentHash	string	SHA-256 of the text. It changes when the content changes.
markdown	string	Main content as Markdown, with tables and code blocks.
text	string	Main content as plain text.
html	string	Cleaned content HTML (saveHtml).
htmlUrl	string (URL)	Link to the stored HTML file (saveHtmlAsFile).
links	array	All links on the page (saveLinks).
fileLinks	array	Document links: url, type, text.
chunks	array	RAG chunks: id, index, text, charCount, tokenEstimate, headings.
crawl	object	requestedUrl, loadedUrl, loadedTime, referrerUrl, depth, httpStatusCode, source.
metadata	object	canonicalUrl, title, description, author, keywords, languageCode, publishedAt, modifiedAt, image, robots, openGraph, jsonLd, headers.
truncated	boolean	Present when a very large page was shortened to fit the dataset's row limit.

The run also saves a CRAWL_REPORT record in its key-value store, listing URLs that were not delivered and why, plus how many pages were skipped as duplicates.

Need more web data?

• Sitemap Extractor: list every URL in a website's sitemaps
• Text Splitter & Chunker: chunk text you already have
• PDF to Text Extractor: get the text from the PDFs listed in fileLinks
• Website Change Monitor: get alerted when a page changes

FAQ

How does billing work? Pay per event: a small fee per run start and a fee per page delivered. Pages that fail, return an error, are blocked by the site, have no readable content or are skipped as duplicates are not charged. Set a maximum cost per run and the crawl stops there, storing only what your budget covers. See the Pricing tab for current rates. Platform usage is listed there too.

How do I crawl only part of a website? Use restrictToStartPath to stay under the start URL's path, or includeUrlGlobs and excludeUrlGlobs for exact patterns such as https://example.com/docs/**.

Which crawler type should I use? Start with http: it is faster and costs less. If a page comes back with no content, the run tells you, and you can switch crawlerType to browser for that site. Browser mode never switches on by itself.

What chunk size should I use for RAG? chunkSize 2000 with chunkOverlap 200 is a good start (roughly 500 tokens per chunk). Chunks follow the page's headings, and each one records its heading path so you can add context to embeddings.

Can it download PDFs? It captures links to PDFs and other documents in fileLinks, with their anchor text. It does not download the files or read their contents. Pair it with a PDF actor for that.

How do I re-crawl only what changed? Set onlyCrawlPagesModifiedSince to the date of your last crawl. Pages dated earlier are skipped and not charged. Pages with no date on the site are delivered. You can also compare contentHash between runs.

Does it remove duplicates? Yes. Pages found while crawling are dropped when they repeat a delivered canonical URL or identical content. Start URLs are always delivered. Set ignoreCanonicalUrl to keep everything.

Why did some pages fail? Open CRAWL_REPORT in the run's key-value store. It gives a reason per URL, for example http-404, blocked-by-site or no-readable-content-without-javascript. Sites that block automated visitors are not worked around: those pages are reported and not charged.

How much memory should I set? The default suits both modes. For http runs you can lower it to reduce platform usage. In our tests a 300-page http crawl peaked under 700 MB, so 1 GB was enough and cost less. Keep the default for browser runs.

What happens if a run is interrupted? The crawl keeps its progress, so a resumed run continues without delivering or charging for finished pages again.

Is it legal to crawl websites? This actor collects publicly available pages. You are responsible for following each site's terms and the laws that apply to your data, especially for personal data. You can enable respectRobotsTxtFile.

Can I connect this to other apps and services? The Website Content Crawler Pro can be connected with almost any cloud service or web app thanks to integrations on the Apify platform. It works with Make, Zapier, Slack, Airbyte, GitHub, Google Drive and many more, plus the Apify API, JavaScript/Python clients and MCP. Or use webhooks to trigger an action whenever a run finishes.

Feedback

How did it go? Good or bad, a review helps other users decide and tells us what to improve. Something broken or missing? Open an issue and we'll look at it.

Related Actors

• Website Content Crawler: the lightweight text-only crawler
• Web to Markdown: convert single pages to Markdown
• Website Contact Scraper: emails, phones and social links from any site
• Bulk URL Checker: status, redirects and metadata for a list of URLs

Notes

Original clean-room implementation. Returns only public data; you are responsible for compliance with each website's terms. Not affiliated with any site it crawls.