# Substack Scraper: Posts, Comments & Authors (`doggo/substack-scraper-posts-comments-authors`) Actor
Scrape any Substack publication: post archives, article text, comments, author profiles and subscriber signals. Search across newsletters and export structured data for research, monitoring and AI datasets. No browser. Output to CSV, JSON or Excel.
- **URL**: https://apify.com/doggo/substack-scraper-posts-comments-authors.md
- **Developed by:** [Doggo](https://apify.com/doggo) (community)
- **Categories:** Marketing, News, Social media
- **Features:** Limited permissions
- **Stats:** 11 total users, 5 monthly users, 98.8% runs succeeded, 0 bookmarks
- **User rating**: 5.00 out of 5 stars
## Pricing
from $3.00 / 1,000 posts
This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.
Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event
## What's an Apify Actor?
An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.
## How to integrate an Actor?
If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.
For examples already wired to this Actor's own input schema, see the [API](#api) section below.
Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).
# README
## Substack Scraper
**Scrape any Substack newsletter, post, author, or comment — fast, cheap, and at scale.**
This Apify actor extracts structured data from Substack publications via their public JSON API. No browser, no JavaScript rendering, no login required. Built for newsletter research, content monitoring, author discovery, competitive intelligence, and LLM training datasets.
### What you can scrape
- **Substack posts** — title, subtitle, full HTML and plain-text body, word count, publish date, tags, cover image, paywall status, reactions, comment count, restacks
- **Substack publications** — name, subdomain, custom domain, description, logo, category, language, subscriber count (when public), founding plan
- **Substack authors** — profile, handle, bio, photo, the publications they write for, the publications they subscribe to
- **Substack comments** — full nested comment threads, author handles, publish dates, reactions, reply depth
Works with any Substack URL: `https://*.substack.com`, custom domains (`https://stratechery.com`), individual post URLs, `https://substack.com/@handle` author profiles, and `https://open.substack.com/pub/...` share links.
### Why use this Substack scraper
- **Pay only for data, not for browser time** — no Playwright, no rendering overhead, no per-minute compute billing. You pay per result, and failed requests are never charged.
- **Full archives, not just the front page** — paginates through the entire publication archive until the very first post.
- **Clean, typed output** — one dataset with a `type` field (`post` / `publication` / `author` / `comment`) and per-type table views, so you can export straight to BI tools, CSV, JSON, Excel, or Google Sheets.
- **No duplicates, no surprises** — every post is delivered exactly once, limits are enforced even across platform restarts, and proxy rotation is handled for you.
### Common use cases
- **Newsletter research** — download the full archive of a competitor's Substack for content analysis, topic clustering, or SEO research
- **Content monitoring** — schedule a daily run with `maxPostsPerPublication: 5` to capture new posts from a tracked list of newsletters and pipe to Slack or email
- **Author discovery and lead generation** — crawl author profiles to map who writes for which publications, then export handles for outreach
- **LLM training data** — bulk-extract long-form Substack content (with word counts and metadata) for fine-tuning datasets
- **Competitive intelligence** — track subscriber counts, post frequency, paywall strategy, and engagement metrics (reactions, comments, restacks) across a competitor set
- **Academic and journalism research** — gather statements, essays, and commentary from Substack writers with citable timestamps
- **Archiving and backup** — export your own Substack publication before a migration
### Input
| Field | Type | Default | Description |
|---|---|---|---|
| `startUrls` | array of URLs | — | Substack publication, post, or author URLs. Leave empty only when using Discovery mode |
| `mode` | `posts` / `publication` | `posts` | What to pull for each publication URL |
| `maxPostsPerPublication` | integer | 50 | Cap per publication. `0` = entire archive. Lower = cheaper |
| `includeContent` | boolean | true | Fetch each post's full HTML body |
| `includeComments` | boolean | false | Fetch comments for each post (each comment is a separate result) |
| `onlyFreePosts` | boolean | false | Skip paid / subscriber-only posts in archives |
| `searchQuery` | string | — | Filter the publication archive by keyword |
| `discoveryMode` | `none` / `leaderboard` / `search` | `none` | Auto-discover many publications without providing URLs |
| `discoveryQuery` | string | — | Keyword for `search` discovery |
| `maxPublicationsToDiscover` | integer | 25 | Cap on discovered publications. Lower = cheaper |
| `maxConcurrency` | integer | 5 | Parallel requests |
#### Discovery mode — scrape many publications without a list
If you don't have a list of specific newsletters, turn on **Discovery mode** and the actor will find publications for you:
- **Top publications** (`leaderboard`) — seeds from 5 curated top Substacks and expands through each publication's recommendations until the limit is hit
- **Search** (`search`) — same expansion, plus your `discoveryQuery` keyword filters every discovered publication's archive
Each discovered publication is then scraped using the same `mode` / `maxPostsPerPublication` settings as `startUrls`, so you can go from zero URLs to a full corpus in one run. Discovery is **off by default** — a discovery run scrapes many publications and produces a correspondingly large dataset.
```json
{
"discoveryMode": "search",
"discoveryQuery": "AI",
"maxPublicationsToDiscover": 50,
"mode": "posts",
"maxPostsPerPublication": 20,
"includeContent": true
}
```
### Example input
```json
{
"startUrls": [
{ "url": "https://www.thefitzwilliam.com" },
{ "url": "https://noahpinion.substack.com" },
{ "url": "https://substack.com/@mattyglesias" }
],
"mode": "posts",
"maxPostsPerPublication": 100,
"includeContent": true,
"includeComments": false
}
```
### Output
All records land in the run's dataset with a `type` discriminator (`post`, `publication`, `author`, `comment`). The Output tab offers per-type table views (**Posts**, **Publications**, **Authors**, **Comments**); for exports, filter on the `type` field to split record types into separate files.
#### Post record
```json
{
"type": "post",
"id": 123456,
"title": "Why newsletters won",
"slug": "why-newsletters-won",
"url": "https://example.substack.com/p/why-newsletters-won",
"publication": "example",
"publicationName": "The Example",
"publishedAt": "2026-02-01T14:00:00Z",
"audience": "everyone",
"isPaid": false,
"author": "Jane Author",
"authors": [{ "id": 99, "name": "Jane Author", "handle": "janeauthor" }],
"bodyHtml": "...
",
"bodyText": "...",
"wordcount": 1842,
"reactionCount": 213,
"commentCount": 42,
"restacks": 18,
"postTags": ["media", "business"]
}
```
#### Publication record
```json
{
"type": "publication",
"id": 42,
"name": "The Example",
"subdomain": "example",
"customDomain": null,
"url": "https://example.substack.com",
"description": "A newsletter about newsletters.",
"categoryName": "Business",
"totalSubscribers": 48211,
"paidSubscribers": 1203,
"createdAt": "2022-06-14T09:12:00Z"
}
```
#### Author record
```json
{
"type": "author",
"id": 99,
"name": "Jane Author",
"handle": "janeauthor",
"profileUrl": "https://substack.com/@janeauthor",
"bio": "Writing about media.",
"photoUrl": "https://.../photo.jpg",
"publications": [{ "publicationName": "The Example", "subdomain": "example", "role": "admin" }],
"subscriptions": [{ "publicationName": "Noahpinion", "subdomain": "noahpinion" }]
}
```
#### Comment record
```json
{
"type": "comment",
"id": 55512,
"postId": 123456,
"postSlug": "why-newsletters-won",
"postTitle": "Why newsletters won",
"publication": "example",
"parentId": null,
"depth": 0,
"body": "Great piece.",
"authorName": "A Reader",
"authorHandle": "areader",
"publishedAt": "2026-02-01T16:30:00Z",
"reactionCount": 4
}
```
### How to scrape Substack (step-by-step)
1. **Click "Try for free"** at the top of this page — you'll be taken to the Apify console.
2. **Paste your target URLs** into the Start URLs field. Examples:
- A publication: `https://stratechery.com` or `https://noahpinion.substack.com`
- A single post: `https://example.substack.com/p/some-post`
- An author profile: `https://substack.com/@handle`
- A share link: `https://open.substack.com/pub/astralcodexten/p/some-post`
3. **Set `maxPostsPerPublication`** — start with `10` for a test, then bump it (or set `0` for the whole archive).
4. **Click "Start"**. When the run completes, open the **Output** tab to browse results or hit **Export** for CSV / JSON / Excel.
### FAQ
**How am I charged?** Per record in your results — each post, publication, author, and comment counts as one result. Failed or retried requests are never charged, and you'll never receive the same post twice. Control your bill with `maxPostsPerPublication`, `includeComments`, and `maxPublicationsToDiscover`; you can also set a maximum budget for any run in the Apify Console.
**Does it scrape paywalled posts?** Paid posts are listed with metadata and the free preview text; full paid bodies require a subscriber login, which this scraper does not use. Enable `onlyFreePosts` to skip them entirely.
**How many comments will a post produce?** Whatever the thread holds — popular posts can carry hundreds of comments, each delivered (and charged) as its own result. Leave `includeComments` off unless you need them.
**Will it get blocked?** No setup needed on your side — proxy rotation, retries, and rate-limit handling are built in.
**Can I schedule it?** Yes — use Apify Schedules for daily/weekly monitoring runs, and connect the dataset to Google Sheets, webhooks, or the API for delivery.
### Use with AI agents (MCP)
This Actor works as a tool for AI agents via the [Apify MCP server](https://mcp.apify.com). Add it to Claude, Cursor, or any MCP-compatible assistant with this server URL:
```text
https://mcp.apify.com?tools=doggo/substack-scraper-posts-comments-authors
```
Then just ask something like *"Scrape the latest 20 posts from noahpinion"* — the agent fills in the input, runs the Actor, and reads the results from the dataset. Standard pay-per-result pricing applies to the calling account.
# Actor input Schema
## `startUrls` (type: `array`):
Substack URLs to scrape: publication homepages (subdomain or custom domain), individual post URLs, author profiles (https://substack.com/@handle), and open.substack.com share links. Leave empty only if you enable Discovery mode below.
## `mode` (type: `string`):
What to scrape for each publication URL. 'posts' = publication metadata + all posts in the archive. 'publication' = publication metadata only. Post and author URLs are always scraped as posts/authors regardless of this setting ('author' is accepted as an alias of 'publication').
## `maxPostsPerPublication` (type: `integer`):
Limit the number of posts scraped per publication (lower = cheaper runs). Set 0 to scrape the ENTIRE archive — large publications can have thousands of posts. Leaving the field empty uses the default of 50.
## `includeContent` (type: `boolean`):
Fetch each post's full HTML + plain text body. Adds one request per post but gives you the actual article content.
## `includeComments` (type: `boolean`):
Fetch the full comment thread for each post (adds one request per post). Note: every comment becomes a separate result in the dataset, and popular posts can have hundreds of comments.
## `onlyFreePosts` (type: `boolean`):
Skip paid/subscriber-only posts in publication archives (their bodies can't be fully scraped anyway). Post URLs given directly in Start URLs are always scraped.
## `searchQuery` (type: `string`):
Filter each publication archive by a search term. Only posts matching the term are scraped.
## `discoveryMode` (type: `string`):
Auto-discover publications without providing URLs — seeds from 5 curated top publications and expands through each publication's /recommendations page until the limit below is hit. 'Search' additionally filters every discovered publication's archive by the Discovery query. Off by default: a discovery run scrapes MANY publications and produces a correspondingly large (and more expensive) dataset.
## `discoveryQuery` (type: `string`):
Used with 'Search' Discovery mode: keyword used to filter the archive of each discovered publication. Ignored otherwise.
## `maxPublicationsToDiscover` (type: `integer`):
Cap on how many publications Discovery mode will feed into the scraper (the 5 seed publications count toward it; Start URLs do not). Lower = cheaper runs.
## `maxConcurrency` (type: `integer`):
Parallel requests. Higher = faster runs; proxy rotation is handled automatically, but very aggressive settings can hit Substack rate limits.
## Actor input object example
```json
{
"startUrls": [
{
"url": "https://noahpinion.substack.com"
}
],
"mode": "posts",
"maxPostsPerPublication": 50,
"includeContent": true,
"includeComments": false,
"onlyFreePosts": false,
"discoveryMode": "none",
"discoveryQuery": "AI",
"maxPublicationsToDiscover": 25,
"maxConcurrency": 5
}
```
# Actor output Schema
## `results` (type: `string`):
Every scraped record as JSON: posts (full article text, engagement metrics), publications (metadata, subscriber counts), author profiles, and comment threads. Distinguish records by the `type` field.
## `posts` (type: `string`):
Records of type `post`: title, author, publish date, full HTML and plain-text body, word count, reactions, comments, restacks, tags, paywall status.
## `publications` (type: `string`):
Records of type `publication`: name, domain, description, language, subscriber counts (when public), creation date.
## `authors` (type: `string`):
Records of type `author`: name, handle, bio, photo, publications they write for and subscribe to.
## `comments` (type: `string`):
Records of type `comment`: flattened comment threads with body, author, depth, parent id, publish date, and reactions.
# API
You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.
## JavaScript example
```javascript
import { ApifyClient } from 'apify-client';
// Initialize the ApifyClient with your Apify API token
// Replace the '' with your token
const client = new ApifyClient({
token: '',
});
// Prepare Actor input
const input = {
"startUrls": [
{
"url": "https://noahpinion.substack.com"
}
]
};
// Run the Actor and wait for it to finish
const run = await client.actor("doggo/substack-scraper-posts-comments-authors").call(input);
// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
console.dir(item);
});
// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs
```
## Python example
```python
from apify_client import ApifyClient
# Initialize the ApifyClient with your Apify API token
# Replace '' with your token.
client = ApifyClient("")
# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://noahpinion.substack.com" }] }
# Run the Actor and wait for it to finish
run = client.actor("doggo/substack-scraper-posts-comments-authors").call(run_input=run_input)
# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
print(item)
# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start
```
## CLI example
```bash
echo '{
"startUrls": [
{
"url": "https://noahpinion.substack.com"
}
]
}' |
apify call doggo/substack-scraper-posts-comments-authors --silent --output-dataset
```
## MCP server setup
```json
{
"mcpServers": {
"apify": {
"type": "http",
"url": "https://mcp.apify.com/?tools=fetch-actor-details,doggo/substack-scraper-posts-comments-authors"
}
}
}
```
The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).
## OpenAPI specification
Download the OpenAPI definition: https://api.apify.com/v2/actors/9xk2c5cd6ksIc379X/builds/oZuta5b55paEoXFxH/openapi.json