# Hacker News Scraper — Stories, Comments & Users (`junipr/hacker-news-scraper`) Actor

Scrape Hacker News stories, comments, user profiles, Ask HN, Show HN, jobs, and threaded discussions from the public Firebase API.

- **URL**: https://apify.com/junipr/hacker-news-scraper.md
- **Developed by:** [junipr](https://apify.com/junipr) (community)
- **Categories:** News, Developer tools
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.95 / 1,000 item scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Hacker News Scraper — Stories, Comments & Users

Extract stories, comments, user profiles, and search results from Hacker News using the official Firebase API. This actor provides full comment threading with recursive replies, clean markdown output optimized for LLMs, and powerful search via the Algolia HN Search API with no API key required.

### Why Use This Actor?

Unlike web scrapers that parse HTML and break when the layout changes, this actor uses Hacker News's official public APIs. This means reliable, structured data every time with no rate-limit headaches or CAPTCHAs. Comment threads are fetched recursively and returned as nested trees, making it easy to analyze discussions. All text content is converted to clean markdown alongside the original HTML, making the output ready for LLM pipelines, RAG systems, and content analysis.

### Features

- **All feed types**: Top, New, Best, Ask HN, Show HN, and Jobs
- **Full comment threading**: Recursive comment fetching with configurable depth and limits
- **Algolia search**: Full-text search across all HN content with tag filtering
- **User profiles**: Karma, about text, account age, and recent submissions
- **Markdown output**: HTML body content converted to clean markdown for LLM consumption
- **Flexible filtering**: Filter by score, comment count, and sort by score/date/comments
- **Poll support**: Full poll option extraction with scores
- **Zero configuration**: Works out of the box with one top-story metadata row; opt into comments or larger result counts when needed

### Input Examples

#### Scrape Top Stories

```json
{
  "scrapeType": "topStories",
  "maxItems": 30,
  "includeComments": true,
  "maxCommentsPerItem": 50,
  "commentDepth": 5
}
```

#### Search Hacker News

```json
{
  "scrapeType": "search",
  "searchQuery": "machine learning",
  "searchTags": "story",
  "maxItems": 20,
  "minScore": 50
}
```

#### Fetch User Profiles

```json
{
  "scrapeType": "user",
  "usernames": ["pg", "dang", "patio11"]
}
```

#### Fetch Specific Items

```json
{
  "scrapeType": "item",
  "itemIds": [8863, 121003],
  "includeComments": true
}
```

### Output Example

Each story in the dataset looks like this:

```json
{
  "id": 12345678,
  "type": "story",
  "title": "Show HN: A new open-source tool for data extraction",
  "url": "https://example.com/tool",
  "text": null,
  "textMarkdown": null,
  "by": "builder42",
  "score": 287,
  "time": 1710150000,
  "timeISO": "2026-03-11T12:00:00.000Z",
  "descendants": 143,
  "comments": [
    {
      "id": 12345679,
      "type": "comment",
      "by": "techfan",
      "text": "<p>This is really impressive work. The API design reminds me of...</p>",
      "textMarkdown": "This is really impressive work. The API design reminds me of...",
      "time": 1710151000,
      "timeISO": "2026-03-11T12:16:40.000Z",
      "parent": 12345678,
      "replies": []
    }
  ],
  "hnUrl": "https://news.ycombinator.com/item?id=12345678"
}
```

### How It Works

The actor uses two official APIs:

1. **HN Firebase API** (`hacker-news.firebaseio.com/v0/`): Fetches story feeds, individual items, comments, and user profiles. This is the same API that powers the HN website.
2. **HN Algolia API** (`hn.algolia.com/api/v1/`): Provides full-text search across all HN content with filtering by story, comment, poll, or job.

Comment threads are fetched recursively — each item contains a list of child IDs, which are fetched individually and assembled into a nested tree structure. Rate limiting is applied automatically to stay within API limits.

### Pricing

This actor uses pay-per-event pricing at **$1.95 per 1,000 items** scraped in the queued monetization update. Each story, comment batch, user profile, or search result counts as one item. You only pay for what you use — no minimum fees or subscriptions.

### Tips for Best Results

- **Speed vs. completeness**: Set `includeComments: false` for fast metadata-only runs. Comment fetching is the slowest part since each comment requires an individual API call.
- **Large runs**: For 500 stories with comments, expect 5-15 minutes depending on comment density. The actor respects HN API rate limits to avoid being blocked.
- **Search**: The Algolia API is fast and supports standard search operators. Use `searchTags` to filter by content type.
- **Filtering**: Use `minScore` and `minComments` to focus on high-quality, well-discussed content.

#### FAQ

#### How many items can I scrape at once?

You can scrape up to 500 items per run. For story feeds, HN typically returns 200-500 story IDs. The `maxItems` parameter limits how many are processed.

#### Does this actor require an API key?

No. Both the HN Firebase API and the Algolia HN Search API are public and free to use. No authentication is needed.

#### Can I get historical data?

Use the search feature with date filters via the Algolia API. The Firebase API only provides current feed positions (top, new, best) but individual items are available forever by ID.

#### Why are some comments missing?

Deleted and dead (flagged/killed) comments are excluded by default. The `maxCommentsPerItem` and `commentDepth` settings also limit how many comments are fetched per story.

#### How is this different from other HN scrapers?

This actor uses the official APIs instead of web scraping, provides full recursive comment threading, converts HTML to LLM-ready markdown, and supports all HN content types including polls, jobs, and user profiles. Most competitors only scrape the front page HTML.

# Actor input Schema

## `scrapeType` (type: `string`):

What type of content to scrape from Hacker News. Choose a story feed (top, new, best, ask, show, jobs), search for content via Algolia, look up specific users, or fetch individual items by ID.

## `maxItems` (type: `integer`):

Maximum number of stories or items to return. Applies to story feeds and search results. Higher values take longer but return more data.

## `includeComments` (type: `boolean`):

Fetch and include the full comment tree for each story. Disabling this significantly speeds up scraping when you only need story metadata.

## `maxCommentsPerItem` (type: `integer`):

Maximum number of comments to fetch per story. Comments are fetched in order of the thread tree (top-level first, then replies). Use lower values for faster runs.

## `commentDepth` (type: `integer`):

Maximum depth of nested replies to follow in comment threads. Depth 1 = top-level comments only. Depth 5 = up to 4 levels of nested replies.

## `searchQuery` (type: `string`):

Search query string for finding stories and comments via the Algolia HN Search API. Required when scrapeType is 'search'. Supports standard search operators.

## `searchTags` (type: `string`):

Filter search results by content type. Only applies when scrapeType is 'search'. Leave empty to search all content types.

## `usernames` (type: `array`):

List of HN usernames to look up. Required when scrapeType is 'user'. Returns profile info including karma, about text, and recent submissions.

## `itemIds` (type: `array`):

List of HN item IDs to fetch directly. Required when scrapeType is 'item'. Accepts story, comment, job, poll, or poll option IDs.

## `minScore` (type: `integer`):

Only include items with a score (upvotes) greater than or equal to this value. Useful for filtering out low-quality content.

## `minComments` (type: `integer`):

Only include stories with at least this many comments (descendants). Useful for finding highly-discussed topics.

## `sortBy` (type: `string`):

How to sort the output results. 'score' sorts by upvotes (highest first), 'date' sorts by newest first, 'comments' sorts by most-discussed first.

## Actor input object example

```json
{
  "scrapeType": "topStories",
  "maxItems": 1,
  "includeComments": false,
  "maxCommentsPerItem": 0,
  "commentDepth": 3,
  "sortBy": "score"
}
```

# Actor output Schema

## `results` (type: `string`):

Stories, comments, user profiles, or search results from Hacker News with full metadata and comment threading.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "scrapeType": "topStories",
    "maxItems": 1,
    "includeComments": false,
    "maxCommentsPerItem": 0,
    "commentDepth": 3
};

// Run the Actor and wait for it to finish
const run = await client.actor("junipr/hacker-news-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "scrapeType": "topStories",
    "maxItems": 1,
    "includeComments": False,
    "maxCommentsPerItem": 0,
    "commentDepth": 3,
}

# Run the Actor and wait for it to finish
run = client.actor("junipr/hacker-news-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "scrapeType": "topStories",
  "maxItems": 1,
  "includeComments": false,
  "maxCommentsPerItem": 0,
  "commentDepth": 3
}' |
apify call junipr/hacker-news-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,junipr/hacker-news-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/H3p2lSkUJGsN4Ucx5/builds/WZgfGlu3mBTpKyRhS/openapi.json