# Website to Markdown Crawler for RAG and LLMs (`ucchiii/website-markdown-crawler`) Actor

Crawl a website from start URLs and get each page as clean Markdown for RAG, LLM and vector database pipelines: title, description, word count and optional chunks. Follows robots.txt, stays on the site, depth and page limits, no browser. Empty input = free demo.

- **URL**: https://apify.com/ucchiii/website-markdown-crawler.md
- **Developed by:** [Actor Factory](https://apify.com/ucchiii) (community)
- **Categories:** AI, Developer tools, Automation
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.70 / 1,000 markdown pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

**Website to Markdown Crawler: give it a start URL and it follows the links on that site, turning every page into clean Markdown — one dataset row per page with url, title, description, markdown, word count and depth. Built for RAG pipelines, LLM context and vector databases. Plain HTTP, no browser, robots.txt respected.**

| url | title | wordCount | depth | markdown (start) |
|---|---|---|---|---|
| https://docs.apify.com/academy/scraping-basics-javascript | Web scraping basics for JavaScript devs | Academy | Apify Documentation | 997 | 0 | `# Web scraping basics for JavaScript devs` `**Learn how to use JavaScript to extract information from websites in this practical course, starting from the absolute basics.**`… |
| https://docs.apify.com/academy/scraping-with-ai | Web scraping with AI and Apify | Academy | Apify Documentation | 426 | 1 | `# Web scraping with AI and Apify` `**Learn how to use AI to extract information from websites, starting from the absolute basics.**`… |

*(real rows from the free demo run on Apify on 2026-11-12; the full row also has description, statusCode, crawledAt, referrerUrl, lang, charCount, truncated and error)*

**For people feeding websites to AI** — documentation for a support bot, a knowledge base for an internal assistant, a blog for a fine-tuning set, product docs for a coding agent — who want the text of a whole site as tidy Markdown without writing a crawler.

**Try it:** press **Start** with the prefilled example (up to 10 pages of a documentation site, about 1 cent). Clear **Start URLs** and press Start for a **free demo** on 5 pages (rows carry `_demo: true`, not charged).

### What you get

- **One row per page**: `url`, `title`, `description`, `markdown`, `wordCount`, `depth`, `statusCode`, `crawledAt`, plus `referrerUrl` (the page that linked to it), `lang`, `charCount`, `truncated` and `error`.
- **Markdown that keeps the structure**: headings (`#`, `##`), numbered and nested lists, tables as GitHub-style pipe tables, code blocks with their language (` ```python `), bold and italics, and links made absolute.
- **Only the content**: menus, site headers, footers, sidebars, cookie banners, share buttons, newsletter boxes and ad-like blocks are removed; when a page has a `<main>` or `<article>` element, only that is kept. Turn **Main content only** off to keep everything except scripts and styles.
- **Stays on the site**: by default only pages in the same folder as the start URL (`/docs/…` stays in `/docs/`); or the whole host; or the host and its subdomains. Links to other websites are never followed.
- **You set the limits**: max pages (up to 5,000 per run), max link depth, URL patterns to include or skip, and optionally the site's **sitemaps** to find pages no link points to.
- **RAG-ready chunks (optional)**: turn on **Split into chunks** and every row also gets a `chunks` array — the page split along its headings into pieces of about the size you choose, each with `headingPath` (`["Install the SDK", "Steps"]`), `charCount` and `approxTokens`. Code blocks are never cut in the middle. Chunks cost nothing extra.
- **Any language**: the charset comes from the HTTP header, a byte-order mark or `<meta charset>`, so Japanese, Chinese, Cyrillic and other pages come out right. `wordCount` counts each Chinese, Japanese or Korean character as one word.

### Example: what a page becomes

A documentation page with a site header, a cookie banner, a sidebar, an ad block, share buttons and a footer around its content becomes (the test page in `tests/`, exact output of the converter):

````markdown
## Install the SDK

Use **pip** to install it. See the [client reference](https://docs.example.com/v2/api/client) or
[PyPI](https://pypi.org/project/example/).

### Steps

1. Create a virtual environment
2. Install the package
   - stable
   - beta

```bash
pip install example-sdk
example --version
```

### Options

| Name | Default |
| --- | --- |
| timeout | 30 |
| retries | 3 |
````

*(output of the converter on a sample page from the test suite; nothing from the header, banner, sidebar, ad, share buttons or footer is left)*

### How to use

1. Put one or more pages in **Start URLs** — usually the top of the docs or blog you want (up to 100 start URLs). A sitemap URL such as `https://example.com/sitemap.xml` also works.
2. Choose **Max pages** (default 100) and **Max link depth** (default 5). Optional: **Scope**, **Only URLs matching** / **Skip URLs matching**, **Read sitemaps**, **Split into chunks**.
3. Press **Start**, or call the Actor from your code:

```json
{
  "startUrls": [{ "url": "https://docs.example.com/guide/" }],
  "maxPages": 300,
  "maxDepth": 4,
  "scope": "folder",
  "excludeUrlPatterns": ["**/changelog/**", "**?page=*"],
  "splitIntoChunks": true,
  "chunkSize": 1500,
  "chunkOverlap": 200
}
```

**URL patterns**: `**` matches anything, `*` matches anything except `/`, and a pattern without `*` matches when the URL contains it. `https://example.com/docs/**` keeps only the docs; `**/tag/**` skips tag pages; `/login` skips any URL containing `/login`. Start URLs are always fetched; pages that do not pass the patterns are not fetched, so links on them are not followed.

### Output fields

- `url` — the page that was read, after redirects (no `#fragment`, no `utm_` tracking parameters). Two URLs that end on the same page give one row.
- `title` — the `<title>` tag, else `og:title`, else the first `<h1>`. `description` — the meta description, else `og:description`.
- `markdown` — the page content as Markdown; `wordCount`, `charCount`; `truncated` is true when **Max characters per page** cut it (or the page was over 5 MB).
- `depth` — 0 for start URLs and sitemap pages, 1 for pages they link to, and so on. `referrerUrl` — where the link was found.
- `statusCode`, `crawledAt` (UTC), `lang`, and `chunks` when chunking is on.
- `error` — `null` on success, otherwise `blocked_by_robots`, `robots_txt_unreachable`, `crawl_delay_too_long`, `blocked_private_address`, `invalid_url`, `timeout`, `dns_failure`, `connection_error`, `tls_error`, `http_404` (or another `http_<status>`), `too_many_redirects`, `redirect_loop`, `redirected_out_of_scope`, `not_html` or `no_content` (an HTML page without text, for example one built entirely by JavaScript).
- **SUMMARY** record: pages fetched, pages turned into Markdown, rows charged, pages skipped by robots.txt, errors by code, duplicates, and whether the run stopped at the page limit or the maximum charge.

To load the result into a vector database, read the dataset items and embed either `markdown` (one document per page) or each entry of `chunks` (with `url` and `headingPath` as metadata).

### Pricing

Pay per event: **$0.001 per Markdown page** — a crawled page turned into Markdown. That is **$1.00 per 1,000 pages** on the free plan; with Store discounts it is $0.90 (BRONZE), $0.80 (SILVER) and **$0.70 per 1,000 (GOLD)**.

**Not charged:** pages skipped because of robots.txt, fetch errors (404, 5xx, timeouts, blocked addresses), redirects off the site, non-HTML files (PDFs, images, downloads), pages without text content, and the free demo (empty input). Chunks are included in the page price.

On the Apify Free plan, the $5 of monthly credit covers about 5,000 pages a month. Set a **maximum charge per run** to cap spend: the crawler stops fetching new pages when it is reached.

### Polite crawling

- **robots.txt is always followed** — read once per host, with the rules for `website-markdown-crawler` if the site names it, otherwise the rules for `*` (longest matching rule wins, `*` and `$` supported). If robots.txt cannot be reached (server error), the host is not crawled. A `Crawl-delay` is honored; hosts asking for more than 60 seconds between requests are skipped.
- `<meta name="robots" content="nofollow">` and `rel="nofollow"` links are not followed.
- At most **2 connections per host** and at least **0.5 s between requests** to the same host, whatever the concurrency. One retry after 429/502/503/504, a timeout or a dropped connection.
- At most **5 MB of HTML** read per page and **5,000 pages** per run. Links to images, PDFs, archives, media and other files are not fetched at all; a response that turns out not to be HTML is not read.
- No login, no cookies, no forms, no JavaScript. The User-Agent names this Actor.
- **Safe by design**: addresses inside private networks (localhost, 10.x, 192.168.x, link-local such as 169.254.169.254, IPv6 loopback/unique-local) are never contacted — every redirect hop is checked before it is followed.

Only crawl sites you are allowed to access, and check each site's terms before reusing its content.

### Limits

- HTTP only: pages that build their text with JavaScript after loading come back empty (`no_content`, not charged).
- Up to 5,000 pages and 100 start URLs per run. For bigger sites, split the work by folder (one run per section) with **Only URLs matching**.
- Pages are processed in parallel, so the dataset order is roughly breadth-first, not strictly the order of links.

### FAQ

**Why did it stop after a few pages?** Check **Scope** (the default stays in the start URL's folder), **Max link depth**, your URL patterns, and the SUMMARY record (`robotsBlocked`, `byError`). Turning on **Read sitemaps** often finds pages that menus built by JavaScript hide.

**Do I pay for pages blocked by robots.txt or broken links?** No. Only rows with Markdown are charged.

**Can I get only the text for embeddings?** Yes: keep **Main content only** on and **Keep images** off (both defaults), and turn on **Split into chunks** if your pipeline wants pieces rather than whole pages.

**Which libraries does it use?** beautifulsoup4 (MIT), lxml (BSD-3-Clause), markdownify (MIT), httpx (BSD-3-Clause) and the Apify SDK (Apache-2.0).

### More tools from the same developer

- [HTML to Markdown Converter for LLM and RAG](https://apify.com/ucchiii/html-to-markdown) — clean Markdown from URLs, raw HTML or datasets for LLM and RAG
- [RAG Chunker](https://apify.com/ucchiii/rag-chunker) — split scraped pages into chunks for RAG and vector stores
- [Sitemap URL Extractor](https://apify.com/ucchiii/sitemap-urls) — list every URL of a website from its sitemaps
- [Broken Link Checker](https://apify.com/ucchiii/broken-link-checker) — find broken links on web pages: status of every link found
- [Link Preview API](https://apify.com/ucchiii/link-preview-api) — link previews for URL lists: Open Graph and Twitter Card title, description, image, site name and favicon

# Actor input Schema

## `startUrls` (type: `array`):

Where the crawl starts: full URLs with http:// or https://. The crawler follows links from these pages and stays on the same site (see Scope). A sitemap URL (for example https://example.com/sitemap.xml) is read as a list of pages. Up to 100 start URLs. Leave empty for a free demo (not charged).

## `maxPages` (type: `integer`):

The most pages fetched in this run, from 1 to 5,000 (default 100). The crawl stops when it is reached. Pages skipped because of robots.txt are not counted and not charged.

## `maxDepth` (type: `integer`):

How many clicks away from a start URL the crawler goes, from 0 to 20 (default 5). 0 = only the start URLs (and sitemap pages); 1 = also the pages they link to.

## `scope` (type: `string`):

Which links are followed. Same folder: https://example.com/docs/intro crawls only pages under https://example.com/docs/. Whole host: every page on example.com. Host and subdomains: also blog.example.com, docs.example.com and so on. Links to other websites are never followed.

## `includeUrlPatterns` (type: `array`):

Optional. Only pages whose URL matches one of these patterns are fetched (start URLs always are). \*\* matches anything, \* matches anything except /; a pattern without \* matches when the URL contains it. Example: https://example.com/docs/\*\* or /blog/. Pages that do not match are not fetched, so their links are not followed either.

## `excludeUrlPatterns` (type: `array`):

Optional. Pages whose URL matches one of these patterns are not fetched. Same pattern rules as above. Example: **/tag/** or /login or \*\*?page=\*.

## `useSitemaps` (type: `boolean`):

Also read the site's sitemaps (the Sitemap lines of robots.txt, else /sitemap.xml) and crawl the pages listed there that are inside the scope and patterns. Useful for pages no link points to.

## `mainContentOnly` (type: `boolean`):

Remove menus, headers, footers, sidebars, cookie banners, share buttons and ad-like blocks, and keep the main or article element when the page has one. Scripts and styles are always removed.

## `keepImages` (type: `boolean`):

Keep images as !\[alt]\(absolute URL) in the Markdown. Off by default, which is usually best for LLM text. Images are never downloaded.

## `maxCharsPerPage` (type: `integer`):

Cut each page's Markdown at this length, at a paragraph boundary where possible (the row gets truncated: true). 0 = no limit.

## `splitIntoChunks` (type: `boolean`):

Add a chunks array to each row: the Markdown split by headings into pieces of about Chunk size characters, with headingPath, charCount and approxTokens. Code blocks are not cut in the middle. Chunks are not charged separately.

## `chunkSize` (type: `integer`):

Target size of each chunk in characters, from 200 to 20,000 (default 1,500, roughly 375 tokens). Used only when Split into chunks is on.

## `chunkOverlap` (type: `integer`):

Characters repeated from the end of one chunk at the start of the next, to keep context (default 200, at most half the chunk size). Used only when Split into chunks is on.

## `concurrency` (type: `integer`):

How many pages are processed in parallel, from 1 to 20 (default 5). Whatever you set, one host gets at most 2 connections at a time and at least 0.5 s between requests (longer if its robots.txt asks for a Crawl-delay).

## `timeoutSecs` (type: `integer`):

Timeout for each request, in seconds, from 1 to 60 (default 20). A page that keeps sending slowly is stopped after 3 times this value.

## `userAgent` (type: `string`):

Optional custom User-Agent header. The default names this Actor. robots.txt rules are always read for the name website-markdown-crawler and for \*.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    }
  ],
  "maxPages": 10,
  "maxDepth": 2,
  "scope": "folder",
  "useSitemaps": false,
  "mainContentOnly": true,
  "keepImages": false,
  "maxCharsPerPage": 0,
  "splitIntoChunks": false,
  "chunkSize": 1500,
  "chunkOverlap": 200,
  "concurrency": 5,
  "timeoutSecs": 20
}
```

# Actor output Schema

## `pages` (type: `string`):

One item per page: url, title, description, markdown, wordCount, depth, statusCode, crawledAt, referrerUrl, lang, charCount, truncated, error, and chunks when chunking is on.

## `summary` (type: `string`):

Record SUMMARY: pages fetched, pages turned into Markdown, rows charged, pages skipped by robots.txt, errors by code, duplicates, and whether the run stopped at the page limit or the maximum charge.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
        }
    ],
    "maxPages": 10,
    "maxDepth": 2,
    "maxCharsPerPage": 0,
    "chunkSize": 1500,
    "chunkOverlap": 200,
    "concurrency": 5,
    "timeoutSecs": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("ucchiii/website-markdown-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.apify.com/academy/web-scraping-for-beginners" }],
    "maxPages": 10,
    "maxDepth": 2,
    "maxCharsPerPage": 0,
    "chunkSize": 1500,
    "chunkOverlap": 200,
    "concurrency": 5,
    "timeoutSecs": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("ucchiii/website-markdown-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    }
  ],
  "maxPages": 10,
  "maxDepth": 2,
  "maxCharsPerPage": 0,
  "chunkSize": 1500,
  "chunkOverlap": 200,
  "concurrency": 5,
  "timeoutSecs": 20
}' |
apify call ucchiii/website-markdown-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ucchiii/website-markdown-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/gqmUIhcTxx5iOKQsk/builds/Hm3zYWgYbEmirVxjQ/openapi.json