# Article/News Extractor (`tropical_prune/article-news-extractor`) Actor

CAPABILITIES: extract_article, extract_metadata, detect_language, clean_text, batch_urls.
INPUT: URLs (single or array) of articles/news pages.
OUTPUT: structured JSON with title, author, date, content, language, word_count.
FORMATS: json, markdown, text.
PRICING: PPE $0.001/article.

- **URL**: https://apify.com/tropical_prune/article-news-extractor.md
- **Developed by:** [Bado](https://apify.com/tropical_prune) (community)
- **Categories:** Developer tools
- **Features:** Limited permissions
- **Stats:** 91 total users, 33 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

$1.00 / 1,000 article extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Article & News Extractor

Extract clean, structured article text and metadata from any news site or blog. Built by **Tropical Tools** — structured data extraction APIs optimized for AI agents.

Feed in URLs and get back title, author, publication date, full article content, language, tags, word count, reading time, and token estimates. The article extractor is purpose-built for RAG pipelines, content analysis, and AI agent workflows that need reliable, clean text from the web.

### What Does It Do?

Article & News Extractor takes any article or blog URL and returns structured, clean content stripped of ads, navigation, sidebars, and other noise. It uses readability-based extraction combined with JSON-LD and Schema.org metadata parsing to pull the most accurate data possible from each page.

Every extraction returns:

- **Title** — The article headline
- **Author** — Byline attribution when available
- **Publication date** — Parsed and normalized to ISO 8601
- **Content** — Full article body, cleaned of HTML cruft
- **Language** — Detected language code (ISO 639-1)
- **Tags/categories** — Topic tags and section labels from the source
- **Word count** — Total words in the extracted content
- **Reading time** — Estimated minutes to read
- **Token estimate** — Approximate token count for LLM context planning
- **Paywall detection** — Flags articles behind paywalls (without bypass)

### Why Use This Actor?

Most web scrapers return raw HTML or half-broken text full of navigation elements and ad copy. This news scraper is different:

- **RAG-optimized output** — Clean text with metadata fields that map directly to vector DB schemas. No post-processing needed before chunking and embedding.
- **Paywall detection** — Identifies paywalled content upfront so your pipeline doesn't ingest truncated articles into your knowledge base.
- **Language detection** — Automatic language identification lets you route multilingual content to the right embedding model or translation step.
- **3 output formats** — Get results as structured JSON, clean Markdown, or plain text depending on your downstream needs.
- **Batch processing** — Pass hundreds of URLs in a single run. The actor processes them concurrently and returns results as they complete.

### Features

- **Readability extraction** — Mozilla Readability-based content isolation that strips ads, navigation, footers, and related-article blocks
- **JSON-LD / Schema.org parsing** — Extracts structured metadata embedded by publishers for maximum accuracy on dates, authors, and categories
- **Multi-platform support** — Tested and optimized for WordPress, Medium, Substack, Ghost, major news outlets (Reuters, AP, BBC, NYT), and thousands of independent blogs
- **Paywall detection** — Detects soft and hard paywalls and flags them in output metadata (does not bypass paywalls)
- **Content deduplication** — Identifies and removes repeated boilerplate text across batch runs
- **Configurable output** — Choose between JSON, Markdown, and plain text formats

### Input Configuration

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `urls` | `array` | required | List of article URLs to extract |
| `outputFormat` | `string` | `"json"` | Output format: `json`, `markdown`, or `text` |
| `includeMetadata` | `boolean` | `true` | Include full metadata (author, date, tags, etc.) |
| `extractStructuredData` | `boolean` | `true` | Parse JSON-LD and Schema.org data from pages |

#### Example Input

```json
{
    "urls": [
        "https://en.wikipedia.org/wiki/Web_scraping",
        "https://example.com/blog/ai-agents-2025"
    ],
    "outputFormat": "json",
    "includeMetadata": true,
    "extractStructuredData": true
}
```

### Output Example

Extracting a Wikipedia article returns structured data like this:

```json
{
    "url": "https://en.wikipedia.org/wiki/Web_scraping",
    "title": "Web scraping",
    "author": "Wikipedia contributors",
    "publishedDate": "2024-11-15T08:22:00Z",
    "content": "Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites. Web scraping software may directly access the World Wide Web using the Hypertext Transfer Protocol or a web browser...",
    "language": "en",
    "tags": ["Web scraping", "Data extraction", "Web harvesting"],
    "wordCount": 4250,
    "readingTimeMinutes": 17,
    "tokenEstimate": 5660,
    "paywallDetected": false,
    "structuredData": {
        "@type": "Article",
        "name": "Web scraping",
        "inLanguage": "en",
        "isPartOf": {
            "@type": "WebSite",
            "name": "Wikipedia"
        }
    },
    "extractedAt": "2025-03-15T12:00:00Z"
}
```

### Cost Estimation

Each successfully extracted article costs **$0.001** (one-tenth of a cent). Failed extractions (404 errors, unreachable URLs) are not charged.

| Volume | Cost | Cost per 1K articles |
|--------|------|---------------------|
| 100 articles | $0.10 | $1.00 |
| 1,000 articles | $1.00 | $1.00 |
| 10,000 articles | $10.00 | $1.00 |
| 100,000 articles | $100.00 | $1.00 |

### Use Cases

- **RAG / vector DB ingestion** — Extract and chunk articles for retrieval-augmented generation pipelines. Clean text with metadata makes embedding and retrieval more accurate.
- **News monitoring** — Track coverage across dozens of publications. Feed URLs from RSS feeds or news APIs and get structured, comparable output.
- **Content analysis** — Analyze word counts, reading levels, topic tags, and publication patterns across large content sets.
- **AI agent reading** — Give your AI agent the ability to read and understand web articles. The token estimate field helps with context window planning.
- **Competitive intelligence** — Monitor competitor blogs, press releases, and thought leadership content in structured form.

### FAQ

#### Does it bypass paywalls?

No. The actor detects paywalled content and flags it in the output (`paywallDetected: true`), but it does not bypass, circumvent, or work around any paywall mechanisms. You will receive whatever content is publicly accessible on the page.

#### What sites are supported?

The article extractor works with any standard HTML page that contains article content. It has been optimized for WordPress, Medium, Substack, Ghost, and major news platforms including Reuters, AP, BBC, The Guardian, and The New York Times. Sites with heavy JavaScript rendering may require additional processing time.

#### How does language detection work?

Language is detected using a combination of HTML `lang` attributes, metadata tags, and content-level analysis. The actor returns an ISO 639-1 language code (e.g., `en`, `es`, `fr`, `de`, `ja`). Detection is accurate for all major languages and most minor ones.

#### Does it handle AMP pages?

Yes. When an AMP version of a page is detected, the actor extracts content from the AMP markup. If you provide an AMP URL directly, it will be processed normally. The actor prefers canonical (non-AMP) versions when both are available, as they typically contain richer metadata.

#### Can I use it with RSS feeds?

The actor accepts direct article URLs, not RSS feed URLs. However, it pairs well with RSS feed actors — use an RSS parser to get article URLs, then pass those URLs to this actor for full content extraction. This is a common pattern for news monitoring pipelines.

#### What happens if a URL returns a 404 or is unreachable?

Failed URLs are reported in the output with an error status and message. They are not charged. The actor continues processing remaining URLs in the batch without stopping.

# Actor input Schema

## `urls` (type: `array`):

List of article/news page URLs to extract. Also accepts a single URL string.

## `outputFormat` (type: `string`):

Format: json (structured), markdown (with YAML frontmatter), or text (plain, for LLM context).

## `includeMetadata` (type: `boolean`):

Include metadata (author, date, tags, language) in output.

## `extractStructuredData` (type: `boolean`):

Extract JSON-LD and Schema.org structured data from articles.

## Actor input object example

```json
{
  "urls": [
    "https://en.wikipedia.org/wiki/Web_scraping",
    "https://en.wikipedia.org/wiki/Artificial_intelligence"
  ],
  "outputFormat": "json",
  "includeMetadata": true,
  "extractStructuredData": false
}
```

# Actor output Schema

## `url` (type: `string`):

The URL of the extracted article.

## `title` (type: `string`):

Article title.

## `author` (type: `string`):

Article author name.

## `publishedDate` (type: `string`):

ISO 8601 publication date.

## `description` (type: `string`):

Article summary or meta description.

## `content` (type: `string`):

Clean article body text (plain text format).

## `markdown` (type: `string`):

Article content formatted as Markdown.

## `language` (type: `string`):

Detected language code (ISO 639-1).

## `publisher` (type: `string`):

Publication or news outlet name.

## `tags` (type: `string`):

Article tags or categories.

## `wordCount` (type: `string`):

Number of words in the article body.

## `readingTimeMinutes` (type: `string`):

Estimated reading time in minutes at 200 wpm.

## `tokenEstimate` (type: `string`):

Estimated LLM token count for the content.

## `paywalled` (type: `string`):

Whether the article appears to be behind a paywall.

## `leadImage` (type: `string`):

URL of the article's main image.

## `section` (type: `string`):

Article section or category from the publication.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://en.wikipedia.org/wiki/Web_scraping",
        "https://en.wikipedia.org/wiki/Artificial_intelligence"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("tropical_prune/article-news-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://en.wikipedia.org/wiki/Web_scraping",
        "https://en.wikipedia.org/wiki/Artificial_intelligence",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("tropical_prune/article-news-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://en.wikipedia.org/wiki/Web_scraping",
    "https://en.wikipedia.org/wiki/Artificial_intelligence"
  ]
}' |
apify call tropical_prune/article-news-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,tropical_prune/article-news-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/WXmNdFcSSC8v9kMt5/builds/QPW3rPIjhshVpdECq/openapi.json