# HTML to Markdown Converter - Web Page to Markdown (LLM-ready) (`gochujang/web-to-markdown`) Actor

HTML to Markdown converter. Convert any URL to Markdown, plain text, or structured JSON. Extracts title, author, date, tags, word count, and reading time. CSS selector support for partial-page extraction. Absolute link normalization and image ALT text preservation. LLM-ready for RAG pipelines.

- **URL**: https://apify.com/gochujang/web-to-markdown.md
- **Developed by:** [Hojun Lee](https://apify.com/gochujang) (community)
- **Categories:** Developer tools
- **Features:** Limited permissions
- **Stats:** 3 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Web Page → Markdown Converter

> **Web scraper to Markdown** — convert any **URL to Markdown** (**webpage to text**, **HTML to Markdown**) with structured metadata (title, author, date, lang, image, tags). **Uses trafilatura — the same library Common Crawl uses.** LLM-ready output. Batch up to 500 URLs. **$0.005 per URL.**
> **Keywords:** web to markdown · HTML to markdown · URL to markdown · content extraction · LLM content scraper · RAG pipeline · trafilatura scraper · web page to text

***

### ⚡ Run in 30 seconds

Click **Start** with default settings — fetches the example URL and returns clean Markdown text plus title, author, publication date, language, and image metadata, all ready to pipe into an LLM. No API key needed.

***

### Input Parameters

| Parameter | Type | Default | Description |
|---|---|---|---|
| `urls` | array | `[]` | List of URLs to convert. Each is billed separately. |
| `url` | string | ``| Single URL (used when 'urls' is empty). |
| `includeComments` | boolean | `false` | Include reader comments / discussion sections. |
| `includeTables` | boolean | `true` | Render HTML tables as Markdown tables. |
| `deduplicate` | boolean | `true` | Drop repeated boilerplate (nav, footer, ads). |
| `userAgent` | string |`` | Custom UA string. Default looks like a desktop browser. |

***

### Why this exists

Most LLM pipelines need clean article-body text — but raw HTML is 60-90% boilerplate (nav, footer, ads, JS, related stories). Existing solutions:

- **Browserless / Puppeteer**: complex setup, $30+/mo
- **Mercury Parser**: deprecated
- **Diffbot**: $299/mo minimum
- **Readability.js**: requires running Node

This actor wraps **trafilatura** — the gold-standard Python library used by Common Crawl and most LLM training pipelines — into a one-call API. Pass a URL list, get clean Markdown + metadata back.

***

### What you get per row

| Field | Example | Notes |
|---|---|---|
| `url` | `https://...` | input URL |
| `ok` | `true` | did extraction succeed |
| `title` | `Bitcoin — Wikipedia` | from `<title>` or og |
| `author` | `Wikipedia contributors` | |
| `description` | `Bitcoin is a cryptocurrency...` | |
| `date_published` | `2025-12-01` | |
| `language` | `en` | auto-detected |
| `sitename` | `Wikipedia` | |
| `tags` | `["cryptocurrency", "blockchain"]` | |
| `categories` | `["Technology"]` | |
| `image` | `https://...` | hero image |
| `markdown` | `# Bitcoin\n\nBitcoin is...` | clean body |
| `char_count` | `48230` | |
| `word_count` | `7842` | |

***

### Quick start

#### Single URL

```json
{
  "url": "https://en.wikipedia.org/wiki/Bitcoin"
}
```

#### Batch of URLs

```json
{
  "urls": [
    "https://techcrunch.com/article-1",
    "https://www.theverge.com/article-2",
    "https://www.wired.com/article-3"
  ],
  "includeTables": true,
  "deduplicate": true
}
```

#### Custom User-Agent (some sites require it)

```json
{
  "url": "https://...",
  "userAgent": "Mozilla/5.0 (compatible; YourBot/1.0; +https://yourdomain.com/bot)"
}
```

***

### Pricing

**Pay-Per-Event**: `$0.005 per URL processed.`

| Run | URLs | Cost |
|---|---|---|
| Single article | 1 | $0.005 |
| Batch of 100 | 100 | $0.50 |
| Daily crawl of 1K URLs | 1000 | $5.00 |

Vs Diffbot ($299/mo), Mercury ($199/mo for similar tier), this is **40-60x cheaper** for typical volumes.

***

### Common pipeline patterns

#### Feed to Claude / GPT for summarization

```bash
## 1. Extract clean text
curl -X POST "https://api.apify.com/v2/acts/gochujang~web-to-markdown/runs?token=$T" \
  -d '{"url":"..."}'

## 2. Pipe markdown to Claude
curl -X POST https://api.anthropic.com/v1/messages \
  -d "{\"messages\":[{\"role\":\"user\",\"content\":\"Summarize: $MARKDOWN\"}]}"
```

#### RSS-style aggregator

1. [Sitemap URL Discovery](https://apify.com/gochujang/sitemap-url-discovery) to get all article URLs
2. Filter by lastmod (recent only)
3. This actor to convert each to Markdown
4. Store in your DB / Notion / Obsidian

#### Personal read-it-later

Schedule this actor with your "saved articles" Google Sheet → get clean markdown into Obsidian / Logseq daily.

***

### Use cases

1. **LLM input prep** — Clean text for RAG / fine-tuning / summarization
2. **Content curation** — Newsletter / digest aggregation
3. **SEO research** — Compare clean content across competitors
4. **Archiving** — Read-it-later in Markdown format
5. **Translation pipelines** — Strip boilerplate before sending to MT

***

### Data source / engine

- **Engine**: [trafilatura](https://trafilatura.readthedocs.io/) — actively maintained, used by Common Crawl
- **Fallback**: Returns `ok: false` with error message if a page can't be extracted (paywall, JS-heavy SPA without SSR, etc.)

***

### Limitations

- **JS-only sites**: Pages that render entirely in client-side JS may return empty markdown. For those, use a browser-rendering actor (Playwright/Puppeteer-based).
- **Paywalls**: This actor doesn't bypass paywalls.
- **Comments / discussion sections**: Off by default; enable with `includeComments: true`.

***

### Related actors (same author)

- [HTML Metadata Extractor](https://apify.com/gochujang/html-metadata-extractor) — Just metadata (OG, Twitter, JSON-LD) without article body
- [Sitemap URL Discovery](https://apify.com/gochujang/sitemap-url-discovery) — Find all URLs to feed into this actor
- [PDF Text Extractor](https://apify.com/gochujang/pdf-text-extractor) — PDF version
- [JSON Schema Generator](https://apify.com/gochujang/json-schema-generator)

***

### Feedback

A short review helps content/AI engineers find it: [Leave a review on Apify Store](https://apify.com/gochujang/web-to-markdown#reviews)

# Actor input Schema

## `urls` (type: `array`):

List of URLs to convert. Each is billed separately.

## `url` (type: `string`):

Single URL (used when 'urls' is empty).

## `markdownOutputFormat` (type: `string`):

Controls the format of the extracted content. 'markdown' (default) returns GitHub-flavoured Markdown. 'json' returns a structured node tree (heading, paragraph, code_block, image, table, list, blockquote). 'text' strips all Markdown syntax and returns plain prose.

## `contentSelector` (type: `string`):

Optional CSS selector. Only the first matching element is converted. Example: 'article.main-content' or '#content'. Leave empty to use the full page body.

## `includeComments` (type: `boolean`):

Include reader comments / discussion sections.

## `includeTables` (type: `boolean`):

Render HTML tables as Markdown tables.

## `deduplicate` (type: `boolean`):

Drop repeated boilerplate (nav, footer, ads).

## `userAgent` (type: `string`):

Custom UA string. Default looks like a desktop browser.

## `extractImages` (type: `boolean`):

When enabled, adds an 'images' array to each result containing \[{url, alt, title}] for every <img> tag found on the page. Duplicates and relative URLs are resolved.

## `extractLinks` (type: `boolean`):

When enabled, adds a 'links' array to each result containing \[{href, text, is_internal}] for every anchor tag. Fragment-only, javascript:, mailto:, and tel: links are excluded. 'is_internal' is true when the link stays on the same domain.

## Actor input object example

```json
{
  "urls": [
    "https://en.wikipedia.org/wiki/Bitcoin"
  ],
  "url": "",
  "markdownOutputFormat": "markdown",
  "contentSelector": "",
  "includeComments": false,
  "includeTables": true,
  "deduplicate": true,
  "userAgent": "",
  "extractImages": false,
  "extractLinks": false
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://en.wikipedia.org/wiki/Bitcoin"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("gochujang/web-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://en.wikipedia.org/wiki/Bitcoin"] }

# Run the Actor and wait for it to finish
run = client.actor("gochujang/web-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://en.wikipedia.org/wiki/Bitcoin"
  ]
}' |
apify call gochujang/web-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,gochujang/web-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/3jrGgHqjX1eUgsk7V/builds/ZjdzHnkaoL3Up6P36/openapi.json