# Sitemap URL Extractor — Every URL in a sitemap.xml as JSON (`eliai/sitemap-url-extractor`) Actor

Extracts every URL from sitemap.xml files, sitemap indexes (recursive) or a bare domain via robots.txt, with lastmod and priority. Priced per file: 100 sitemap files cost $0.50. Runs with limited permissions, so AI agents can call and pay for it through Apify's x402 agentic payments (experimental).

- **URL**: https://apify.com/eliai/sitemap-url-extractor.md
- **Developed by:** [Broke to Built](https://apify.com/eliai) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Features:** Limited permissions
- **Stats:** 3 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.00 / 1,000 parsed sitemap files

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor: extract URLs from sitemap.xml, priced per file not per URL

**Give it a sitemap URL, a sitemap index, or just a bare domain** - get back every URL the site
declares, with `lastmod`, `changefreq`, `priority` and the sitemap each one came from. Nested
sitemap indexes are followed automatically. Duplicates are removed across every file.

The pricing shape is the whole point: **$0.005 per sitemap file parsed, plus $0.015 per 1,000 URLs
extracted** (rounded up per run). Actors that bill per URL turn an ordinary 5,000-URL sitemap into
$2.50 to $150. Here it is **$0.08**.

***

### What you get

- **One record per URL**: `loc`, `lastmod`, `changefreq`, `priority`, `sourceSitemap`.
- **Sitemap index recursion** - index files are followed up to 5 levels deep, or turn it off with
  `followNested: false`.
- **Domain mode** - pass `example.com` and it discovers sitemaps from the `Sitemap:` lines in
  `robots.txt`, falling back to `/sitemap.xml`.
- **Deduplicated** across every nested file, so a URL listed in two sitemaps appears once.
- **Fail-soft**: a dead or malformed sitemap returns `{ "ok": false, "error": ... }` and is **never
  charged**; the rest of the run keeps going.

### Input

```json
{
  "url": "https://www.example.com/sitemap.xml",
  "urls": ["example.org", "https://example.net/sitemap_index.xml"],
  "maxUrls": 5000,
  "followNested": true
}
```

| Field | Type | Default | Notes |
|---|---|---|---|
| `url` | string | - | A sitemap URL, a sitemap-index URL, or a bare domain |
| `urls` | array of strings | - | Several of any of the above in one run |
| `maxUrls` | integer | `5000` | Hard cap on URLs returned, and therefore on the volume charge (max 50,000) |
| `followNested` | boolean | `true` | Follow child sitemaps inside a sitemap index |

Hard caps per run: **50,000 URLs, 200 sitemap files, 5 levels** of index nesting.

### Output

One dataset item per URL (real run, 2026-08-07):

```json
{
  "loc": "https://www.google.com/intl/en/about/",
  "lastmod": "2026-07-30",
  "changefreq": null,
  "priority": null,
  "sourceSitemap": "https://www.google.com/about/sitemap.xml"
}
```

A sitemap that cannot be fetched or parsed comes back as a diagnostic record, uncharged:

```json
{ "sitemap": "https://example.com/dead.xml", "ok": false, "error": "HTTP 404" }
```

Filter on `ok !== false` if you only want URLs.

***

### Example

A real run on 2026-10-31 with the input this page's form is prefilled with:

```json
{
  "url": "https://www.apify.com/sitemap.xml",
  "urls": [
    "https://apify.com/sitemap.xml"
  ]
}
```

It finished in 114 seconds and returned 5000 results. It was charged for 3 `sitemap-parsed` events ($0.015 on the Free plan) and 5 `urls-extracted-1000` events ($0.075 on the Free plan). The first result (long text and lists cut for display):

```json
{
  "loc": "https://apify.com/",
  "lastmod": null,
  "changefreq": null,
  "priority": null,
  "sourceSitemap": "https://apify.com/sitemap/pages.xml"
}
```

### Pricing

**$0.005 per sitemap file parsed** (one event per index or urlset file that actually fetched and
parsed) **plus $0.015 per started 1,000 URLs** extracted across the run. Fetch errors, broken XML
and documents that are not sitemaps are recorded free.

Worked examples:

| Job | Files parsed | URLs | Cost |
|---|---|---|---|
| One small sitemap | 1 | 300 | $0.02 |
| One large sitemap | 1 | 5,000 | $0.08 |
| An index plus 5 children | 6 | 5,000 | $0.105 |
| A big site, capped | 12 | 50,000 | $0.81 |

#### Honest comparison

Measured from each actor's live pricing on **2026-08-15**. Note that store actors price in
**tiers** by your Apify plan - the "Free tier" column is the list price a new buyer sees, and the
"Best paid tier" column is the cheapest that actor ever gets. This actor has a single flat price
with no tiers.

| Actor | Price shape (free tier) | 5,000 URLs, free tier | 5,000 URLs, their best tier |
|---|---|---|---|
| **This actor** | **$0.005/file + $0.015 per 1,000 URLs** | **$0.08** | **$0.08** |
| `apify/sitemap-extractor` | $0.0005 per URL | $2.50 | $0.50 |
| `crawlerbros/sitemap-url-extractor` | $0.005 start + $0.002 per URL | $10.01 | $5.01 |
| `onescales/sitemap-url-extractor` | $0.00005 start + $0.03 per URL | $150.00 | $13.00 |

**Where they win:** on a tiny sitemap. Ten URLs costs $0.005 at `apify/sitemap-extractor` versus
$0.02 here, because our first per-1,000 event fires at URL number one. Against that actor's free
tier the crossover is about **40 URLs**; against its cheapest paid tier, about **200**. If your job
is a handful of URLs and nothing else, that is the cheaper tool and you should use it.

**Where this wins:** every site above a couple of hundred pages, and the gap widens from there -
over 30x cheaper at 5,000 URLs, and it never scales with URL count the way per-URL pricing does. The
volume event exists so a 50,000-URL run does not cost us more to serve than it earns; a listing
that quietly loses money on its own happy path does not stay up, and we would rather say that than
surprise you later.

### When NOT to use this

- **You want the page content, not the URL list.** This reads sitemaps only. Feed the `loc` values
  into [Webpage Text Extractor](https://apify.com/eliai/webpage-text-extractor) for the content.
- **The site has no sitemap.** There is no crawler here, no link following. If `robots.txt` and
  `/sitemap.xml` come up empty, you get a diagnostic record, not a discovered URL list.
- **Gzipped sitemaps.** `.xml.gz` files are not decompressed yet - they come back as an
  unrecognized document, uncharged.
- **Plain-text or RSS sitemap variants.** Only XML `<urlset>` and `<sitemapindex>` are parsed.
- **You need to know whether those URLs work.** A sitemap is a claim, not a guarantee. Check the
  URLs with [Broken Link Checker](https://apify.com/eliai/broken-link-checker) or
  [Redirect Chain Checker](https://apify.com/eliai/redirect-chain-checker).
- **Sites that block bots at the CDN.** Those fetches fail, and are free.

***

### Limits

- `maxUrls`: at most 50000 (default 5000), at least 1.
- Default run settings: 1024 MB memory, 3600 s timeout. Change them per run under Options if a large input needs more.
- Time: the prefilled example finished in 114 s on 2026-10-31; run time grows with the number of inputs.
- What it is not built for: see *When NOT to use this* on this page.

### FAQ

**How do I get a list of every URL on a website?**
If the site publishes a sitemap, pass the domain - `example.com` - and this finds the sitemaps via
`robots.txt`, follows any index files, and returns one record per declared URL. That is the
complete list *the site itself claims*, which is the closest thing to a definitive URL inventory
without crawling.

**What is a sitemap index, and do I have to handle it myself?**
No. A sitemap index is a sitemap of sitemaps - large sites split their URLs across many files
because a single sitemap is capped at 50,000 URLs. Pass the index; child sitemaps are fetched
recursively up to 5 levels and the results are merged and deduplicated.

**Can I pass just a domain instead of the sitemap URL?**
Yes. Anything that does not look like a sitemap path is treated as a domain: `robots.txt` is read
for `Sitemap:` directives, and `/sitemap.xml` is tried as a fallback.

**What does lastmod actually tell me?**
The date the site claims the page last changed. It is self-reported and often wrong or templated,
but when it is maintained it is the cheapest possible signal for "what changed since my last
crawl" - sort by it and re-fetch only the top.

**Why is this priced per sitemap file instead of per URL?**
Because a sitemap is one HTTP request whether it holds 10 URLs or 50,000, so per-URL pricing
charges you for work that did not happen. The small per-1,000 volume event covers the real cost of
writing that many result rows. See the comparison table above for what the difference adds up to.

**What happens if a sitemap is broken or returns a 404?**
You get a record with `ok: false` and the reason, the run continues with the other inputs, and
that file is not charged.

**Does it respect robots.txt?**
It *reads* `robots.txt` to discover sitemaps. It does not crawl the site, so there is nothing to
disallow - it fetches only the sitemap files themselves.

**How many URLs can one run return?**
50,000, from at most 200 sitemap files, at up to 5 levels of index nesting. Use `maxUrls` to set a
lower cap when you want to bound the spend.

***

### For agents and automation

- **Capability:** enumerate every URL a site declares in its sitemaps
- **Required input:** `url` (sitemap URL or domain) or `urls` (array)
- **Returns:** one JSON record per URL; records with `ok: false` are diagnostics, not URLs
- **Bounded:** `maxUrls`, a 200-file cap and a 5-level depth cap keep every run finite
- **Side effects:** none - GET requests only

```bash
curl -X POST "https://api.apify.com/v2/acts/EliAI~sitemap-url-extractor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "content-type: application/json" \
  -d '{"url":"example.com","maxUrls":5000}'
```

Agents: connect [Apify MCP](https://mcp.apify.com) and call the `EliAI/sitemap-url-extractor` tool.

### Related actors

- [Sitemap Analyzer](https://apify.com/eliai/sitemap-analyzer) - validate a sitemap's health, not just its contents
- [Robots.txt Analyzer](https://apify.com/eliai/robots-txt-analyzer) - see what you are actually blocking
- [Webpage Links Extractor](https://apify.com/eliai/webpage-links-extractor) - every link on a single page
- [Broken Link Checker](https://apify.com/eliai/broken-link-checker) - find the dead ones
- [Webpage Text Extractor](https://apify.com/eliai/webpage-text-extractor) - turn those URLs into clean text

Built by Broke to Built.

### Changelog

- 2026-11-01: Page text only: a clearer page heading, an Example section from a real run and a Limits section. Code, output and price are unchanged.

### For AI agents

This Actor is built to be called by software, not just by people.

- **Mount it directly as an MCP tool** — no Store search, no ranking, just this one tool:
  `https://mcp.apify.com/?actors=eliai/sitemap-url-extractor`
- **Or call it over HTTP** and get the results in the same request:
  `POST https://api.apify.com/v2/acts/eliai~sitemap-url-extractor/run-sync-get-dataset-items`
- **Costs are predictable before you call.** Pricing is pay-per-event (see Pricing above), so an agent can budget a run in advance instead of discovering the bill afterwards.
- **Send only the field you mean.** If you pass the bulk field, it is used on its own; the single-value field is a fallback, never merged into your request. You are charged for the items you sent and nothing else.

### Who uses it

- SEO auditors pulling a site's full declared URL list to compare against what is indexed
- Scraping engineers seeding a crawler with every product URL from a store's nested sitemaps
- Content teams sorting URLs by `lastmod` to find pages untouched for years
- Migration leads exporting the old site's URLs to build a redirect map
- Competitive analysts counting how many pages a rival publishes per sitemap section

# Actor input Schema

## `url` (type: `string`):

A sitemap.xml URL, a sitemap index URL, or a bare domain (e.g. example.com). For a domain, the actor probes robots.txt and /sitemap.xml automatically.

## `urls` (type: `array`):

Optional. Process several sitemaps or domains in one run.

## `maxUrls` (type: `integer`):

Maximum number of URLs to extract across all (nested) sitemaps.

## `followNested` (type: `boolean`):

When the sitemap is an index, recursively fetch and parse its child sitemaps.

## Actor input object example

```json
{
  "url": "https://www.apify.com/sitemap.xml",
  "urls": [
    "https://apify.com/sitemap.xml"
  ],
  "maxUrls": 5000,
  "followNested": true
}
```

# Actor output Schema

## `results` (type: `string`):

Every item this run produced. Export as JSON, CSV or Excel from the dataset view.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "url": "https://www.apify.com/sitemap.xml",
    "urls": [
        "https://apify.com/sitemap.xml"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("eliai/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "url": "https://www.apify.com/sitemap.xml",
    "urls": ["https://apify.com/sitemap.xml"],
}

# Run the Actor and wait for it to finish
run = client.actor("eliai/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "url": "https://www.apify.com/sitemap.xml",
  "urls": [
    "https://apify.com/sitemap.xml"
  ]
}' |
apify call eliai/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,eliai/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/CM3ynznQ9byFFBTMG/builds/SjADqYGayTS3N1CUu/openapi.json