# AgentReader: Clean Web Content for AI Agents (`f0rty7even/agent-reader`) Actor

Turn any URL (HTML page or PDF) into clean, token-efficient Markdown with metadata, ready for an AI agent or LLM. Auto-detects PDFs and renders JavaScript-heavy pages when needed.

- **URL**: https://apify.com/f0rty7even/agent-reader.md
- **Developed by:** [Michael Yousrie](https://apify.com/f0rty7even) (community)
- **Categories:** AI, Developer tools
- **Features:** Limited permissions
- **Stats:** 3 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.00 / 1,000 read (fast path)s

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AgentReader: Clean Web Content for AI Agents

Turn any URL into clean, token-efficient, cited content in a single call. AgentReader fetches a web page or a PDF, strips the navigation, ads, and boilerplate, and returns the main content as tidy Markdown plus structured metadata. It is built for AI agents and LLM pipelines, and it works as an MCP tool out of the box.

### Why AgentReader

Most scrapers hand an agent a wall of raw HTML: menus, cookie banners, footers, and scripts that waste tokens and bury the answer. AgentReader is tuned for how agents actually consume the web:

- **Token efficient.** Aggressive main-content extraction, plus an optional hard `maxTokens` budget so an agent can cap its own context cost. Every result reports its exact token count.
- **Reliable.** A fast HTTP path first, with an automatic headless-browser fallback for JavaScript-heavy pages and pages that soft-block a plain request. The agent gets content instead of an empty response.
- **One shape for HTML and PDF.** Pages and PDFs return the same structured result, so your calling code never has to branch on the source type.

### What it does

Give AgentReader one or more URLs. For each one it returns:

- `content`: clean Markdown (or plain text) of the main body
- `title` and `metadata` (author, date, description, site name, language)
- `tokenCount` and whether the content was `truncated` to your budget
- `sourceType` (`html` or `pdf`) and `renderedWith` (`http` or `browser`)

Failed reads come back with a clear `error` code (`blocked`, `timeout`, `not_extractable`, `too_large`, `not_found`) and are never billed.

### Inputs

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `urls` | array of strings | required | The URLs to read. HTML pages and PDFs are both auto-detected. |
| `maxTokens` | integer | `0` | Hard truncation budget per URL. `0` means no cap. |
| `includeLinks` | boolean | `true` | Keep hyperlinks in the Markdown. |
| `format` | string | `markdown` | `markdown` or `text`. |
| `renderJs` | string | `auto` | `auto` tries a fast fetch and falls back to a browser when a page is empty or blocked. `never` disables the browser. `always` forces it. |

### Example

Input:

```json
{
  "urls": ["https://en.wikipedia.org/wiki/Model_Context_Protocol"],
  "maxTokens": 4000
}
```

Each dataset item contains the clean Markdown, the metadata, and the token count, ready to feed straight into an agent or a RAG index.

### Use as an MCP tool

AgentReader is available as an MCP tool, so assistants and agent frameworks that speak the Model Context Protocol can call it directly to read the web on demand. Point your MCP client at the Apify MCP endpoint with your Apify token and the `read` tool appears automatically.

### Pricing

AgentReader uses pay-per-event pricing. You are charged only for a successful read. Reads that need the headless-browser fallback are billed at a separate rate because they use more compute. Failed reads are free.

# Actor input Schema

## `urls` (type: `array`):

One or more URLs to read. HTML pages and PDFs are both supported and auto-detected.

## `maxTokens` (type: `integer`):

Hard truncation budget per URL. 0 means no cap. Lets an agent bound its context cost.

## `includeLinks` (type: `boolean`):

Preserve hyperlinks in the extracted Markdown.

## `format` (type: `string`):

Content format returned per URL.

## `renderJs` (type: `string`):

auto tries a fast HTTP fetch and falls back to a browser only when the result is empty or blocked. never disables the browser. always forces it.

## `proxyConfiguration` (type: `object`):

Proxies to route requests through. Apify Proxy is recommended, as it reduces blocks on sites that reject datacenter or repeated requests.

## Actor input object example

```json
{
  "urls": [
    "https://docs.apify.com/platform"
  ],
  "maxTokens": 0,
  "includeLinks": true,
  "format": "markdown",
  "renderJs": "auto",
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://docs.apify.com/platform"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("f0rty7even/agent-reader").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://docs.apify.com/platform"] }

# Run the Actor and wait for it to finish
run = client.actor("f0rty7even/agent-reader").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://docs.apify.com/platform"
  ]
}' |
apify call f0rty7even/agent-reader --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,f0rty7even/agent-reader"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/eRK78XQDoGRasLWdN/builds/rlmZsShau2n5E0bce/openapi.json