# arXiv Scraper - Papers, Authors & Abstracts (`logiover/arxiv-paper-scraper`) Actor

Scrape arXiv preprints in bulk. Extract paper title, abstract, authors, categories, publication and update dates, abstract page and PDF links from any search query to CSV/JSON. No API key, no login.

- **URL**: https://apify.com/logiover/arxiv-paper-scraper.md
- **Developed by:** [Logiover](https://apify.com/logiover) (community)
- **Categories:** Developer tools, AI, Agents
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.50 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## arXiv Scraper - Papers, Authors & Abstracts

![Apify Actor](https://img.shields.io/badge/Apify-Actor-00A67E?logo=apify\&logoColor=white) ![No API key](https://img.shields.io/badge/No%20API%20key-required-2ea44f) ![Pay per result](https://img.shields.io/badge/Pricing-Pay%20per%20result-1C7ED6) ![Export](https://img.shields.io/badge/Export-JSON%20%7C%20CSV%20%7C%20Excel-F59E0B)

**Scrape arXiv preprints in bulk. Extract paper title, abstract, authors, categories, publication and update dates, abstract page and PDF links from any search query to CSV/JSON. No API key, no login.**

***

### What does the arXiv Scraper do?

This Actor turns any arXiv search into a structured dataset. You give it by category or by topic or author, it walks the result pages one after another, and it writes one clean row per paper into your dataset — ready to export as JSON, CSV or Excel, or to pull straight from the Apify API.

arXiv publishes an official, keyless Atom API and the Actor queries it directly. A single broad query reports hundreds of thousands of matches, and the offset parameter walks them in blocks, so depth is bounded by real data rather than by guesswork. Pagination is followed automatically until it runs out of results, hits your page limit or hits your **Max items** cap, whichever comes first. Every row is de-duplicated across the whole run, so you are never billed twice for the same paper.

There is **no API key, no login and no browser** involved. That keeps runs fast and cheap, and it means you can schedule the Actor without worrying about credentials expiring.

### Who is it for?

- **Research and R\&D teams** tracking publications in a field.
- **AI teams** building corpora of abstracts and metadata.
- **Competitive-intelligence analysts** watching what rivals publish.
- **Librarians and bibliometricians** assembling citation-adjacent datasets.
- **Journalists and analysts** monitoring a scientific topic.

### Use cases

- Export every paper matching a topic and analyse how it evolved by year.
- Track which authors publish most in a category.
- Build an abstract corpus for embedding or classification.
- Monitor a category weekly for newly posted preprints.
- Collect PDF links for a bulk download pipeline.

### Why use this arXiv Scraper?

- 🔑 **Keyless** — no account, no API token, no cookies to paste.
- 📦 **13 fields per paper** — everything the result page exposes, already typed.
- 📄 **Real pagination** — it walks page after page instead of returning the first screen.
- 🎯 **Precise caps** — **Max items** stops the run exactly where you want it, so the bill is predictable.
- 📊 **Export anywhere** — JSON, CSV, Excel or HTML, plus the Apify API and integrations.
- 💸 **Pay per result** — you pay for rows you actually receive, with no platform fees to calculate.

### What data can you extract?

Every run produces one row per paper, with these fields:

| Field | Type | Description |
|-------|------|-------------|
| `paperId` | string | arXiv identifier including version |
| `title` | string | Paper title |
| `abstract` | string | Full abstract text |
| `authors` | string | Authors, separated by a pipe |
| `primaryCategory` | string | Primary arXiv category |
| `categories` | string | All categories, separated by a pipe |
| `publishedAt` | string | ISO timestamp the first version was posted |
| `updatedAt` | string | ISO timestamp of the latest revision |
| `absUrl` | string | Abstract page URL |
| `pdfUrl` | string | Direct PDF URL |
| `query` | string | Search query this row came from |
| `page` | number | Result block the paper appeared in |
| `scrapedAt` | string | ISO timestamp of extraction |

#### Output example

```json
{
  "absUrl": "https://arxiv.org/abs/2609.19144v1",
  "abstract": "Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and\u2026",
  "authors": "Peter Chen | Xi Chen | Wotao Yin | Tianyi Lin",
  "categories": "cs.CL | cs.AI | cs.LG",
  "page": 1,
  "paperId": "2609.19144v1",
  "pdfUrl": "https://arxiv.org/pdf/2609.19144v1",
  "primaryCategory": "cs.CL",
  "publishedAt": "2026-09-16T17:59:35Z",
  "query": "cat:cs.LG",
  "scrapedAt": "2026-09-17T08:41:04.130Z",
  "title": "A Zeroth-Order Paradigm for LLM Preference Alignment",
  "updatedAt": "2026-09-16T17:59:35Z"
}
```

### How to use

#### Option A — by category

```json
{
  "maxItems": 1000,
  "maxPagesPerSearch": 10,
  "searchQueries": [
    "cat:cs.LG"
  ]
}
```

1. Open the Actor and fill in the category field.
2. Set **Max pages per search** and **Max items** to bound the run.
3. Click **Start**, then export from the **Output** tab.

#### Option B — by topic or author

```json
{
  "maxItems": 500,
  "maxPagesPerSearch": 5,
  "searchQueries": [
    "all:transformer",
    "au:bengio"
  ]
}
```

Paste one or more arXiv URLs into **Start URLs** and the Actor paginates each of them independently.

### Input parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `searchQueries` | array | `–` | arXiv query syntax, e. |
| `maxPagesPerSearch` | integer | `3` | How many result pages to walk for each search. |
| `maxItems` | integer | `100` | Stop after this many papers. |
| `maxConcurrency` | integer | `2` | Parallel requests. |
| `proxyConfiguration` | object | `{"useApifyProxy": true}` | Proxy used to fetch pages. |

### Tips for best results

- Use arXiv's own query syntax: `all:transformer`, `cat:cs.LG`, `au:hinton` all work.
- The API returns up to 100 entries per request, so large exports need few calls.
- Restrict by category (`cat:cs.CL`) to keep a run focused and fast.
- `updatedAt` differs from `publishedAt` when a paper has been revised — useful for tracking activity.
- `pdfUrl` is a direct link, ready for a bulk download step.
- Schedule a weekly run on the same query to build a publication timeline.
- Authors are pipe-separated so the column survives a CSV round trip.
- arXiv asks for courteous use; keep concurrency low and the run will not be throttled.
- Abstracts are full text, so exports can get large — cap results if you only need metadata.
- Pair with the Crossref or OpenAlex scrapers to add citation data.

### Integrations

Send results straight into the tools you already use: **Google Sheets**, **Slack**, **Zapier**, **Make**, **Airtable** or any **Webhook**. You can also **schedule** the Actor to run hourly, daily or weekly and have each run append to the same dataset, which is how you build a price or availability history rather than a one-off snapshot.

### API usage

Run the Actor and collect results from any language. Replace `<YOUR_TOKEN>` with your Apify API token.

**cURL**

```bash
curl -X POST "https://api.apify.com/v2/acts/logiover~arxiv-paper-scraper/run-sync-get-dataset-items?token=<YOUR_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"maxItems": 1000, "maxPagesPerSearch": 10, "searchQueries": ["cat:cs.LG"]}'
```

**Node.js**

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_TOKEN>' });
const run = await client.actor('logiover/arxiv-paper-scraper').call({"maxItems": 1000, "maxPagesPerSearch": 10, "searchQueries": ["cat:cs.LG"]});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

**Python**

```python
from apify_client import ApifyClient

client = ApifyClient('<YOUR_TOKEN>')
run = client.actor('logiover/arxiv-paper-scraper').call(run_input={"maxItems": 1000, "maxPagesPerSearch": 10, "searchQueries": ["cat:cs.LG"]})
for item in client.dataset(run['defaultDatasetId']).iterate_items():
    print(item)
```

### 🤖 AI Agents, MCP & RAG Integration

This Actor is available through the Apify MCP server, so an AI agent can call it as a tool. Point your agent at `https://mcp.apify.com` and it can run the arXiv Scraper on demand — for example: *"Pull the first 500 papers from arXiv and summarise the price distribution."* The agent receives the same structured rows you would get from the UI.

Connect this Actor directly to **Apify AI**, **Claude Desktop**, **Cursor**, **VS Code**, **Windsurf**, **LangChain**, or **LlamaIndex** via the official **Apify MCP Server** — no custom wrapper code needed:

```json
{
  "mcpServers": {
    "apify": {
      "type": "http",
      "url": "https://mcp.apify.com?tools=logiover/arxiv-paper-scraper"
    }
  }
}
```

- **Schema-typed inputs & outputs:** Clean JSON records with explicit `.actor/input_schema.json` and `.actor/output_schema.json` definitions so LLM agents infer parameters and parse fields accurately without HTML noise.
- **Bounded & predictable runs:** Built-in volume caps and pay-per-result budget guards prevent runaway agent tool loops.
- **RAG & automation ready:** Stream run datasets straight into vector stores, n8n, Make, Zapier, or webhook pipelines.

### FAQ

#### Do I need a arXiv account or API key?

No. The Actor reads publicly available pages only. There is nothing to authenticate and no credentials to rotate.

#### How many papers can I get in one run?

As many as the search exposes. Raise **Max pages per search** and **Max items** together; the run stops at whichever limit it reaches first.

#### Why did I get fewer rows than I asked for?

The search ran out of papers. That is normal for narrow queries — broaden the search or add more searches to one run.

#### Are results de-duplicated?

Yes. Each paper is emitted once per run, even when it appears on several pages, so you are never billed twice for the same record.

#### Why are some fields empty?

arXiv does not publish every attribute for every paper. Empty means the source did not show it, not that extraction failed.

#### What export formats are supported?

JSON, CSV, Excel, HTML and RSS from the **Output** tab, plus the Apify API and any integration you connect.

#### How fast is it?

It is pure HTTP with no browser, so a page of results typically takes a second or two. Raise **Max concurrency** carefully — the source rate-limits aggressive crawling.

#### Can I schedule it?

Yes. Use the Apify scheduler to run it on any interval and append each run to the same dataset for time-series analysis.

#### Does it work behind a proxy?

It uses Apify Proxy automatically. You can switch groups or supply your own proxies in **Proxy configuration**.

#### How often does the data change?

arXiv updates continuously. Re-run whenever you need current data; the Actor always reads the live pages, never a cache.

#### Is the output schema stable?

Yes. Field names and types are fixed, so downstream pipelines will not break between runs.

#### What if the site changes its layout?

Open an issue on the **Issues** tab and it gets fixed. The Actor is actively maintained.

### 💬 Support, Issues & Reviews

- **Bug reports & target-site changes:** Hit an unexpected empty result or selector change? [Open a ticket in the Issues tab](https://apify.com/logiover/arxiv-paper-scraper/issues) with your **Run ID** or sample input — we actively monitor issues and ship fixes fast.
- **Feature & field requests:** Need an additional data attribute or filter? [Request it in the Issues tab](https://apify.com/logiover/arxiv-paper-scraper/issues).
- **Enjoying this Actor?** If it saved you engineering time, please consider leaving an honest [review on Apify Store](https://apify.com/logiover/arxiv-paper-scraper/reviews) or bookmarking [logiover/arxiv-paper-scraper](https://apify.com/logiover/arxiv-paper-scraper) — organic developer feedback helps others discover reliable tools and keeps maintenance prioritized.

### Is it legal?

This Actor reads only publicly available pages on arXiv — the same content any visitor sees without logging in. It does not bypass authentication, does not collect private data and does not attempt to defeat access controls. You are responsible for how you use the output: respect the source's terms of service, applicable copyright, and data-protection law such as GDPR where personal data is involved. Scraping public data is generally lawful in the EU and the US, but the responsibility for the downstream use of that data sits with you.

This Actor is an independent tool. It is not affiliated with, endorsed by, or sponsored by arXiv; arXiv and all other trademarks mentioned here are the property of their respective owners.

### Related scrapers

- [Crossref Scraper](https://apify.com/logiover/crossref-scraper) — DOI and citation metadata
- [OpenAlex Scraper](https://apify.com/logiover/openalex-scraper) — Works, authors and institutions
- [PubMed Scraper](https://apify.com/logiover/pubmed-scraper) — Biomedical literature
- [DataCite Scraper](https://apify.com/logiover/datacite-doi-scraper) — Research data DOIs

### How much arXiv data can you extract in one run?

A single run returned **800 papers** in 21 seconds on the example input. That is roughly **2,285 papers a minute**. Widen the search — more queries, more pages, a broader location — and the same run goes deeper.

There is no hard ceiling built into this Actor. What bounds a run is the result limit you set, the breadth of the input you give it, and how much arXiv itself publishes. Because billing is per result, a wider run costs proportionally more and a narrow one costs almost nothing — so it is worth starting small and scaling once the output looks right.

### 📝 Changelog

#### 2026-11-14

- Verified live high-volume benchmark (2026-11-14): extracted **650** structured records in **9.2s** on the production build.
- Hardened internal proxy stall fallback and pay-per-event dataset instance budget protection.

#### 2026-11-11

- Added reliable terminal run status telemetry and structured Apify AI / MCP Server (`mcp.apify.com`) tool configuration.
- Added direct Issues & Reviews support links and refreshed live maintenance verification.

#### 2026-11-01

- Monthly health check (2026-11-01): a live run on the current build completed successfully and returned data. Input options, output fields and pricing are unchanged.
- README refreshed: independence disclaimer added.

#### 2026-10-15

- A run that reaches its maximum cost per run now stops cleanly with its results saved and a status message, instead of ending as aborted.
- A proxy group your account cannot use no longer stops the run at start-up; it falls back to the default Apify Proxy.
- Memory is managed within the run's own limit, which removes a rare out-of-memory crash on large pages.
- The default used when the field is left out for `maxPagesPerSearch` is now 60 (was 3).
- The prefilled example for `maxItems` is now 200 (was 100).
- The default used when the field is left out for `maxItems` is now 4,000 (was 100).

#### 2026-09-23

- Fleet-wide quality audit. Verified end to end against live data and re-checked the input schema, the output columns and the run configuration.
- Output verified on a live run: 100 row(s) across 13 columns, 100.0% of cells populated.
- Run reliability reviewed: 100.0% of public runs succeeded in the last 30 days.
- Input schema, output schema and pricing configuration reviewed.

#### 2026-07-18

- Maintenance & reliability pass — re-verified end-to-end against live data via the Apify API; confirmed the Actor completes successfully and returns non-empty, well-formed results within the 5-minute quality window on its default input.
- Refreshed build so the latest version is current; no change to inputs, output fields or scraping logic.

#### 2026-07-12

- **Runs with empty input** — all fields are now optional (`required: []`). With no input, the actor browses the most recent submissions in a sensible default category (`cs.AI`), newest first, bounded by `maxResults`.
- **Category dropdown** — added a `category` select with a comprehensive list of arXiv subject categories (CS, stat, math, physics, astro-ph, cond-mat, quant-ph, hep, q-bio, q-fin, econ, eess) and human-readable labels. The legacy free-text `categories` array is kept as an advanced option.
- **Sort controls** — `sortBy` relabeled and a new `sortOrder` (descending/ascending) select added. Default sort is now newest submissions first.
- **Correct category filtering** — category filters are now applied via arXiv's `cat:` field syntax inside `search_query` (previous `cat=` query param was a no-op). Date filter now uses arXiv's `submittedDate:[..]` window.
- **Numeric fields** — added `version` (integer) and `authorCount` (integer) as real numbers; dataset view renders them with `format: number`.
- **Robustness** — per-attempt proxy IP rotation with a direct-connection fallback on the final retry, retry on non-200, a ~4-minute time budget that exits gracefully with collected data, and top-level error handling that logs instead of hard-failing the run.
- **Schemas** — added a documented dataset `fields` schema (title + description per field) and refreshed the dataset view.

# Changelog

This Actor's version history is a separate document: https://apify.com/logiover/arxiv-paper-scraper/changelog.md

# Actor input Schema

## `searchQueries` (type: `array`):

arXiv query syntax, e.g. all:transformer, cat:cs.LG, au:hinton.

## `maxPagesPerSearch` (type: `integer`):

How many result pages to walk for each search. Set 0 for no limit.

## `maxItems` (type: `integer`):

Stop after this many papers. Set 0 for no limit.

## `maxConcurrency` (type: `integer`):

Parallel requests. Lower this if the site rate-limits you.

## `proxyConfiguration` (type: `object`):

Proxy used to fetch pages. Defaults to Apify Proxy (automatic). You usually do not need to change this.

## Actor input object example

```json
{
  "searchQueries": [
    "cat:cs.LG"
  ],
  "maxPagesPerSearch": 3,
  "maxItems": 200,
  "maxConcurrency": 2,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

All papers extracted in this run.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchQueries": [
        "cat:cs.LG"
    ],
    "maxPagesPerSearch": 3,
    "maxItems": 200,
    "maxConcurrency": 2
};

// Run the Actor and wait for it to finish
const run = await client.actor("logiover/arxiv-paper-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "searchQueries": ["cat:cs.LG"],
    "maxPagesPerSearch": 3,
    "maxItems": 200,
    "maxConcurrency": 2,
}

# Run the Actor and wait for it to finish
run = client.actor("logiover/arxiv-paper-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchQueries": [
    "cat:cs.LG"
  ],
  "maxPagesPerSearch": 3,
  "maxItems": 200,
  "maxConcurrency": 2
}' |
apify call logiover/arxiv-paper-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,logiover/arxiv-paper-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/SomfKURGxmGPJJeRf/builds/7UkAKdC41nXFAmLbc/openapi.json