# Discourse Forum Scraper (`bindler/discourse-forum-scraper`) Actor

Extract topics and full post text from any Discourse forum. Clean plain-text output for LLM training, RAG corpora and community research.

- **URL**: https://apify.com/bindler/discourse-forum-scraper.md
- **Developed by:** [Neil Sangwaiya](https://apify.com/bindler) (community)
- **Categories:** AI, Developer tools, Social media
- **Features:** Limited permissions
- **Stats:** 2 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Discourse Forum Scraper

Extract every topic and full post text from **any Discourse forum**, as clean plain text. Built for LLM training data, RAG corpora, and community research.

### Use it when you need to

- scrape a Discourse forum to plain text
- export Discourse topics and posts as JSON
- build an LLM training set from a community forum

Discourse powers thousands of developer, product and support communities: Discourse Meta, Rust, Elixir, Home Assistant, OpenAI, Hugging Face, Figma, McNeel, Ubuntu, and countless niche forums. This Actor works on all of them without configuration. Just paste the forum URL.

### What you get

| Field | Description |
|---|---|
| `topicId` | Discourse topic ID |
| `title` | Topic title |
| `url` | Direct link to the thread |
| `category` | Category ID |
| `tags` | Topic tags |
| `createdAt` | When the thread started |
| `lastPostedAt` | Most recent reply |
| `replyCount` | Number of replies |
| `postCount` | Total posts |
| `views` | View count |
| `likeCount` | Likes on the thread |
| `pinned` / `closed` | Thread state |
| `posts[]` | Every post: author, date, plain text, likes, post number |
| `fullText` | The whole thread as one clean text block, ready for embedding |
| `forum` | Source forum |
| `scrapedAt` | ISO timestamp |

### Why `fullText` matters

Discourse returns posts as HTML. Feeding that to a model wastes tokens on markup. This Actor strips it to plain text and joins the thread into a single `fullText` field, so each record drops straight into a vector store or fine-tuning set with no preprocessing.

It also handles Discourse's post pagination. Discourse only returns the first chunk of posts in a topic payload, so most scrapers silently truncate long threads. This one fetches the remaining posts and returns the complete conversation.

### Three modes

- **Latest topics**: the forum's recent activity
- **A category**: pass a category slug like `feature` or `support/33`
- **Search results**: pass any Discourse search query

### Example input

```json
{
  "forumUrl": "https://meta.discourse.org",
  "mode": "latest",
  "maxTopics": 500,
  "includePosts": true,
  "maxPostsPerTopic": 100
}
```

### Example output

```json
{
  "topicId": 1,
  "title": "New to Discourse? Start here!",
  "url": "https://meta.discourse.org/t/new-to-discourse-start-here/1",
  "replyCount": 5,
  "views": 118812,
  "posts": [
    {
      "postNumber": 1,
      "author": "Discourse",
      "createdAt": "2023-05-15T10:00:00.000Z",
      "text": "We're so glad you're here! This is our official community...",
      "likes": 42,
      "isOriginalPost": true
    }
  ],
  "fullText": "Discourse: We're so glad you're here!...",
  "scrapedAt": "2026-09-09T19:19:33.000Z"
}
```

### Cost control

`maxTopics` and `maxPostsPerTopic` cap every run. You are charged per topic returned, so a run's cost is knowable before you start it.

### Notes

- Uses only public, unauthenticated Discourse JSON endpoints. No login required, no private content accessed.
- Rate-limited and retried politely, with backoff on HTTP 429.
- Works on self-hosted Discourse instances as well as hosted ones.

# Actor input Schema

## `forumUrl` (type: `string`):

Base URL of any Discourse forum, e.g. https://meta.discourse.org

## `mode` (type: `string`):

Latest topics, a single category, or search results.

## `categorySlug` (type: `string`):

Only used when mode is 'category'. Example: 'feature' or 'support/33'.

## `searchQuery` (type: `string`):

Only used when mode is 'search'.

## `maxTopics` (type: `integer`):

Stop after this many topics. Controls your cost.

## `includePosts` (type: `boolean`):

Fetch every reply, not just topic metadata. Needed for LLM and RAG use.

## `maxPostsPerTopic` (type: `integer`):

Caps very long threads.

## Actor input object example

```json
{
  "forumUrl": "https://meta.discourse.org",
  "mode": "latest",
  "maxTopics": 100,
  "includePosts": true,
  "maxPostsPerTopic": 50
}
```

# Actor output Schema

## `topics` (type: `string`):

One record per topic, including every post as plain text and a combined fullText field ready for embedding.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "forumUrl": "https://meta.discourse.org"
};

// Run the Actor and wait for it to finish
const run = await client.actor("bindler/discourse-forum-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "forumUrl": "https://meta.discourse.org" }

# Run the Actor and wait for it to finish
run = client.actor("bindler/discourse-forum-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "forumUrl": "https://meta.discourse.org"
}' |
apify call bindler/discourse-forum-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,bindler/discourse-forum-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Jqo4j7TROHGpcOhSh/builds/MmZMyU1dwJLH4IopU/openapi.json