Sitemap Extractor: Get All URLs from Any Website

Sitemap Extractor gets every URL from a website's sitemaps. Give it a domain; it finds the sitemaps via robots.txt, walks sitemap indexes and .gz files, and returns each URL with lastmod. Optionally it checks the HTTP status of each page to catch 404s and redirects. From $0.0003 per URL.

How to use it

1. Enter a domain or a sitemap URL in Websites or sitemap URLs.
2. Optionally add Only URLs matching / Skip URLs matching patterns, a Modified on or after date, or turn on the status check.
3. Click Start and export the URL list as Excel, CSV or JSON.

Why this one

• Finds sitemaps for you: reads Sitemap: lines in robots.txt, then tries the common paths (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, .gz, .txt)
• Walks sitemap indexes of any depth, de-duplicates URLs, and keeps going when one child sitemap is broken (errors are listed in the run summary)
• Reads every format: XML, gzipped .xml.gz, plain-text sitemaps, and RSS/Atom feeds
• Filters before you pay: include/exclude regex patterns, "modified after" date, and a per-site cap, so you're only charged for the URLs you keep
• Optional status check with HEAD and automatic GET fallback for servers that reject HEAD

Output

Field	Example
site	https://docs.apify.com
url	https://docs.apify.com/platform/actors
lastmod, changefreq, priority	2026-10-12, weekly, 0.7
images, hreflang	3, ["en","de"]
sitemap	The sitemap file the URL came from
httpStatus, finalUrl	200 / 301 / 404, and the URL after redirects (status check only)

A summary per site (how the sitemap was found, sitemaps read, URL count, errors) is saved in the run's key-value store.

Use cases

Feeding a clean URL list to a crawler or RAG pipeline, SEO audits (404s and redirects in the sitemap), monitoring competitors' new pages with "modified after", content inventories and site migrations.

Input example

{
  "startUrls": ["docs.apify.com", "https://example.com/sitemap_index.xml"],
  "includePatterns": ["/blog/"],
  "modifiedAfter": "2026-09-01",
  "checkStatus": false
}

How much does it cost to extract all URLs from a sitemap?

Pay per event: you pay only for results.

Example: 10,000 URLs cost $3.00 listed only, or $8.00 with the status check ($0.0008 per checked URL instead of $0.0003). Filtered-out URLs and failed sitemaps are not charged.

Apify also charges its standard run-start fee of $0.00005 per GB of run memory (a fraction of a cent per run). The exact prices are always on the Pricing tab, and you can cap any run with Maximum cost per run.

FAQ

Do I need the sitemap address? No. A domain is enough; it reads robots.txt and tries the common sitemap paths.

Which formats are supported? XML, gzipped .xml.gz, plain-text sitemaps and RSS/Atom feeds.

Do I pay for URLs my filters remove? No. You're only charged for URLs you keep.

More Tidyfold tools

Tool	Use it for
Website Screenshot API: Bulk, Full-Page & Mobile	website screenshot API
HTML to PDF Converter API: URL & HTML to PDF	HTML to PDF
PDF Table Extractor: PDF to Excel, CSV & JSON	PDF to Excel
Invoice & Receipt Parser: PDF to JSON (No AI Key)	invoice parser
PDF OCR: Scanned PDF to Text Extractor	PDF OCR
NPI Registry Scraper: US Healthcare Provider Leads	NPI registry scraper
Contractor License Scraper: Texas, Colorado & Washington	contractor license lookup
New Business Leads: Newly Registered LLCs (CO, NY, OR)	new business leads
Contact Details Scraper: Emails, Phones & Tech Stack	contact details scraper
Bulk WHOIS Lookup: Domain Expiry & Availability (RDAP)	bulk WHOIS lookup

Use it from code or no-code tools

Every run can be started from the Apify API, the Apify Python/JS clients, or connected to n8n, Make, Zapier and Google Sheets through Apify integrations. Results download as JSON, CSV, Excel or XML.

from apify_client import ApifyClient
client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("tidyfold/sitemap-url-extractor").call(run_input={'startUrls': ['docs.apify.com', 'https://example.com/sitemap_index.xml'], 'includePatterns': ['/blog/'], 'modifiedAfter': '2026-09-01', 'checkStatus': False})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

Support

Found a bug or need a field added? Open an issue on the Issues tab. Tidyfold maintains these tools daily and usually replies within 1 business day.