# Website URL Extractor - Get All Site URLs (`lofomachines/urls-extractor`) Actor

Extract every URL from any website automatically — no code needed. This URL extractor crawls pages and parses XML sitemaps into one structured list with metadata (lastmod, priority, changefreq). Filter by keyword, cap results, and export to JSON, CSV, or Excel. Built for SEO audits & migrations.

- **URL**: https://apify.com/lofomachines/urls-extractor.md
- **Developed by:** [Lofomachines](https://apify.com/lofomachines) (community)
- **Categories:** Developer tools, SEO tools, Automation
- **Features:** Limited permissions
- **Stats:** 241 total users, 3 monthly users, 100.0% runs succeeded, 6 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

from $0.20 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website URL Extractor - Get All URLs from Any Website

**Website URL Extractor** automatically **extracts ALL URLs from any website** — combining XML sitemap parsing with recursive crawling to return a complete, structured list of every page URL. Just add one or more start URLs and get back a clean URL list in minutes. Perfect for **SEO audits**, **site migrations**, **content inventories**, and feeding **data pipelines**.

👉 To try it, paste a website URL into `startUrls` and hit **Start**. That's it.

### What does Website URL Extractor do?

This URL extractor discovers and collects every reachable URL on a website. It reads the site's **XML sitemap** when available and also **crawls pages recursively**, so you capture URLs that aren't listed in the sitemap. For each URL it returns any available metadata — **last modified date (`lastmod`)**, **priority**, and **change frequency (`changefreq`)** — and lets you filter results by keyword. No coding required.

It works on any domain, handles large sites (50,000+ URLs), and runs on the Apify platform, so you also get **scheduling, API access, integrations, proxy rotation, and monitoring** out of the box.

### Why use this URL extractor?

- 🔍 **Complete URL discovery** — sitemap parsing + recursive crawling find every page, including orphan pages missing from the sitemap.
- 💨 **Fast and scalable** — extract URLs from whole sites with 50,000+ pages efficiently.
- 📦 **Bulk multi-domain** — pass several start URLs to scrape URLs from multiple websites in one run.
- 🏷️ **Rich metadata** — capture `lastmod`, `priority`, and `changefreq` to track recently updated content.
- 🔎 **Keyword URL filtering** — return only URLs containing your keywords (e.g. `/blog/`, `/product/`).
- 🎯 **Result control** — use `maxResults` to sample or `returnAll` for a full export.
- 🛡️ **Reliable** — automatic retries, proxy support, and loop prevention.

### What data can Website URL Extractor extract?

| Field | Description |
|---|---|
| `url` | The extracted page URL |
| `lastmod` | Last modified date (when available in the sitemap) |
| `priority` | Sitemap priority value of the URL |
| `changefreq` | How often the page is expected to change |

### How do I extract all URLs from a website?

1. Open the Actor and paste one or more website URLs into the **`startUrls`** field.
2. (Optional) Set **`keywords`** to keep only matching URLs, e.g. `["blog"]`.
3. (Optional) Set **`returnAll: true`** for the full list, or use **`maxResults`** to cap output.
4. Click **Start** and wait for the run to finish.
5. Download your URL list as **JSON, CSV, XML, or Excel** from the dataset.

### How much does it cost to extract URLs from a website?

This URL extractor is intentionally lightweight — it parses URL structures without rendering JavaScript unless strictly necessary, keeping platform usage low.

| Site size | Estimated cost |
|---|---|
| Small (< 1,000 URLs) | A few cents per run |
| Medium (~10,000 URLs) | Typically under $1.00 |
| Large (50,000+ URLs) | Scales efficiently; depends on site complexity |

💡 **Tip:** Keep **Apify Proxy** enabled to avoid IP blocking on larger sites.

### Input

Website URL Extractor accepts a JSON input. Only `startUrls` is required — click the **Input** tab for all options.

#### Minimal input

```json
{
  "startUrls": [{ "url": "https://example.com" }]
}
```

#### Full input example

```json
{
  "startUrls": [
    { "url": "https://apify.com" },
    { "url": "https://crawlee.dev" }
  ],
  "proxyConfiguration": { "useApifyProxy": true },
  "returnAll": true,
  "maxResults": 1000,
  "keywords": ["blog", "article"]
}
```

| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
| `startUrls` | Array | ✅ Yes | `[{ "url": "https://apify.com" }]` | One or more websites to extract URLs from |
| `proxyConfiguration` | Object | ❌ No | `{ "useApifyProxy": false }` | Proxy settings for reliable, unblocked access |
| `returnAll` | Boolean | ❌ No | `true` | Extract all URLs and ignore `maxResults` |
| `maxResults` | Integer | ❌ No | `1000` | Max URLs to return. Ignored when `returnAll` is `true` |
| `keywords` | Array | ❌ No | `[]` | Case-insensitive filter — returns only URLs containing **all** listed keywords |

### Output

Results are stored in the **Apify dataset** and can be downloaded as **JSON, CSV, XML, or Excel**, or pulled via the **Apify API**.

```json
[
  {
    "url": "https://example.com/blog/post-1",
    "lastmod": "2026-05-14",
    "priority": "0.8",
    "changefreq": "weekly"
  },
  {
    "url": "https://example.com/products/item-42",
    "lastmod": "2026-04-02",
    "priority": "0.6",
    "changefreq": "monthly"
  }
]
```

### Use cases

- **SEO audit** — extract all URLs to map site architecture and find orphan pages.
- **Site migration** — build a complete page inventory before a redesign or replatform.
- **Content inventory** — list every page to plan content audits and updates.
- **Change monitoring** — track `lastmod` dates to spot recently updated pages.
- **Data pipelines** — feed the URL list into downstream scrapers (HTML scraper, Google Sheets, etc.).
- **Targeted scraping** — use keyword filters to scope extraction to a section like `/blog/`.

### Run Website URL Extractor with the Apify API or AI agents

You can run this URL extractor programmatically via the **Apify API** in any language, schedule recurring runs, or call it from an **AI agent through the Apify MCP server** — making it easy for assistants like Claude or ChatGPT to extract all URLs from a website on demand.

### Build a complete website intelligence pipeline

This URL extractor is step one. Once you have the full URL list, power up your analysis:

| Actor | What it adds |
|---|---|
| [GEO Audit — AI Search Optimization Checker](https://apify.com/lofomachines/geo-technical-audit) | Score each page's visibility in ChatGPT, Perplexity, and Gemini |
| [Website Tech Profiler](https://apify.com/lofomachines/website-tech-profiler) | Detect the tech stack behind any URL — CMS, frameworks, analytics, CDN |
| [Organization Registered Domain & Subdomain Scraper](https://apify.com/lofomachines/organization-registered-domain-subdomain-scraper) | Discover all domains owned by an organization before extraction |
| [Website API & Endpoint Analyzer](https://apify.com/lofomachines/website-api-and-endpoint-analyzer) | Uncover hidden API endpoints behind any extracted URL |

### FAQ

#### How do I get a list of all pages on a website?

Add the website to `startUrls` and run the Actor with `returnAll: true`. It parses the sitemap and crawls the site to return every reachable page URL.

#### Can I extract URLs from an XML sitemap?

Yes. Website URL Extractor reads XML sitemaps automatically and pulls `lastmod`, `priority`, and `changefreq` where available, then supplements them with recursive crawling.

#### Is it legal to extract URLs from a website?

This Actor only collects publicly available page URLs and does not extract private user data. URLs are public information, but you remain responsible for complying with the target site's terms and applicable laws.

#### Why am I getting fewer URLs than expected?

The site's sitemap may be incomplete, some URLs may be blocked by `robots.txt`, or `maxResults` may be limiting output. Set `returnAll: true` for the full list. JavaScript-heavy single-page apps (SPAs) may also expose fewer URLs.

#### What if no URLs are returned?

Check that the `startUrls` are public and not behind a login, and enable `proxyConfiguration` with `useApifyProxy: true` if the site is blocking requests.

### Support

Found a bug or need a feature? Open an issue on the **Issues** tab — feedback is welcome, and custom solutions based on this Actor are available on request.

# Actor input Schema

## `startUrls` (type: `array`):

List of website URLs to extract all pages from. You can add multiple websites for bulk processing. The scraper will automatically discover and extract all available URLs from each website.

## `returnAll` (type: `boolean`):

If enabled, extracts all available URLs regardless of the maxResults setting. When disabled, the maxResults limit will be applied.

## `maxResults` (type: `integer`):

Maximum number of URLs to extract. This limit is ignored if 'Return All URLs' is enabled. Set to 0 for unlimited (same as enabling 'Return All URLs').

## `keywords` (type: `array`):

Filter URLs to only include those containing all specified keywords. For example, if you add 'blog', only URLs containing 'blog' will be returned (e.g., https://example.com/blog/article). You can add multiple keywords - URLs must contain ALL of them.

## `proxyConfiguration` (type: `object`):

Select proxies to be used by the scraper. Recommended for large websites or multiple websites to ensure reliable extraction.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://www.theuselesswebindex.com/"
    },
    {
      "url": "https://crawlee.dev"
    }
  ],
  "returnAll": true,
  "maxResults": 1000,
  "keywords": [
    "blog",
    "article"
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://www.theuselesswebindex.com/"
        }
    ],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("lofomachines/urls-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://www.theuselesswebindex.com/" }],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("lofomachines/urls-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://www.theuselesswebindex.com/"
    }
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call lofomachines/urls-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,lofomachines/urls-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Za1MUlRp0VE4uT4Jg/builds/K9g7rY92gbMPym6g2/openapi.json