# Semrush Scraper - Traffic, Authority, Backlinks & Keywords (`scrapesage/semrush-scraper`) Actor

Scrape Semrush public website overviews: monthly visits, Authority Score, global/country/category rank, organic vs paid traffic, backlinks, referring domains, top keywords with CPC, competitors, traffic by country, device split and AI-assistant referrals.

- **URL**: https://apify.com/scrapesage/semrush-scraper.md
- **Developed by:** [Scrape Sage](https://apify.com/scrapesage) (community)
- **Categories:** SEO tools, Agents, Integrations
- **Features:** Limited permissions
- **Stats:** 29 total users, 14 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$6.00 / 1,000 website overviews

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Semrush Scraper - Traffic, Authority Score, Backlinks & Keywords

> **Disclaimer:** This Actor is an independent tool and is not affiliated with, endorsed by, or sponsored by Semrush Inc. or any of its subsidiaries. All trademarks mentioned are the property of their respective owners. "Semrush" is referenced only to describe the publicly available website this Actor collects data from.

Turn any list of domains into a clean competitive-intelligence table. This actor reads Semrush's **public website-overview pages** and returns the whole page as structured data - monthly visits, Authority Score, global/country/category rank, organic vs paid search traffic, backlink profile, top organic keywords with CPC, competitors, per-country traffic with device splits, and **AI-assistant referral traffic**.

**No Semrush login. No API key. No browser.** Keyless HTTP against pages Semrush publishes for search engines.

### Why this Semrush scraper?

| Data | This actor |
| --- | --- |
| Monthly visits + change % | ✅ |
| Authority Score | ✅ |
| Global / country rank | ✅ + trend (up/down/flat) |
| Category rank + category name | ✅ |
| Pages per visit, time on site, bounce rate | ✅ |
| **Organic vs paid** search traffic, each with change % | ✅ |
| **Backlinks + referring domains**, each with change % | ✅ |
| **6-month history** - visits, organic/paid, backlinks, referring domains | ✅ |
| **Top organic keywords** with position, volume, **CPC**, traffic share | ✅ |
| Organic competitors with relevance + their visits | ✅ |
| Traffic by country with **desktop/mobile share per country** | ✅ |
| Desktop vs mobile visits + 6-month device history | ✅ |
| **AI-assistant referral traffic** (ChatGPT, Perplexity…) | ✅ |
| Referrals in / out (who sends traffic here, where it goes) | ✅ |
| Google penalty risk | ✅ |

**53 fields per domain.** Uncovered domains are reported honestly and **never charged**.

### Use cases

- **Competitor benchmarking** - drop in 50 competitor domains, get one comparable table.
- **Lead qualification** - score prospects by real traffic, Authority Score and backlink profile before you pitch.
- **SEO due diligence** - check a site's organic trend and backlink growth before an acquisition.
- **Agency reporting** - refresh a client + competitor set on a schedule and push to Sheets.
- **AI-visibility tracking** - see how much traffic ChatGPT and friends actually send a domain.

### How to use

1. Add your domains under **Domains** (one per line) - or paste/link a whole list under **Import domains from a file**.
2. Leave everything else at its default and hit **Start**.
3. Export as JSON, CSV or Excel, or pull it from the API.

### Input

```json
{
    "domains": ["booking.com", "stripe.com"],
    "includeKeywords": true,
    "includeCompetitors": true,
    "includeAiTraffic": true,
    "outputFields": [],
    "maxItems": 100
}
```

- **domains** - domains to analyse. Full URLs and Semrush overview links are accepted and reduced to the registrable domain.
- **startUrls** - paste Semrush overview URLs directly. Also accepts the Console's **Link remote text file** option.
- **urlsFromFile** - bulk import: paste the whole list, or give **one link** to a `.txt`/`.csv`, Google Sheet, Drive share or key-value-store record. Blank lines, `#` comments, CRLF, a UTF-8 BOM, quotes and extra CSV columns all parse; duplicates are analysed once.
- **includeHistory / includeKeywords / includeCompetitors / includeCountries / includeAiTraffic / includeDevices / includeReferrals** - toggle each section. All come from the same page load, so turning one off makes the record leaner, not cheaper.
- **outputFields** *(pick-list)* - tick the fields you want and every record is trimmed to exactly those. Nothing to memorise - the dropdown lists all 53 with plain-English labels and their exact key.
- **maxItems** - cap per run.
- **proxyConfiguration** - Apify datacenter by default; Semrush serves these pages cleanly to it.

### Output

One record per domain (`type: "website"`):

```json
{
    "type": "website",
    "domain": "stripe.com",
    "url": "https://www.semrush.com/website/stripe.com/overview/",
    "country": "US",
    "categories": ["Finance"],
    "authorityScore": 79,
    "googlePenaltyRisk": "Low",
    "globalRank": 163,
    "globalRankTrend": "up",
    "countryRank": 164,
    "countryRankName": "United States",
    "categoryRank": 6,
    "categoryRankName": "Finance",
    "visits": 179937086,
    "visitsChangePercent": -0.0757,
    "pagesPerVisit": 2.7837,
    "timeOnSiteSec": 484,
    "bounceRate": 0.6082,
    "organicTraffic": 96442310,
    "organicChangePercent": 0.0312,
    "paidTraffic": 1204882,
    "backlinks": 224518903,
    "backlinksChangePercent": 0.0141,
    "referringDomains": 633826,
    "desktopVisits": 70856331,
    "mobileVisits": 109080755,
    "desktopShare": 0.3938,
    "mobileShare": 0.6062,
    "topKeywords": [
        { "keyword": "stripe", "position": 1, "searchVolume": 1830000, "cpc": 3.11, "trafficShare": 0.0421 }
    ],
    "competitors": [{ "domain": "gocardless.com", "relevance": 0.58, "visits": 3204118 }],
    "trafficByCountry": [
        { "country": "US", "countryName": "United States", "traffic": 52207086, "trafficShare": 0.2901, "desktopShare": 0.3938, "mobileShare": 0.6062 }
    ],
    "aiTraffic": [{ "source": "ChatGPT", "traffic": 227075, "share": 0.0005 }],
    "referralsIn": [{ "host": "github.com", "share": 0.11, "visits": 402118 }],
    "visitsHistory": [{ "date": "2026-06-01", "visits": 179937086 }],
    "dataDate": "2026-06-15",
    "scrapedAt": "2026-08-09T19:44:00.000Z"
}
```

#### What to expect (field coverage)

Measured over a deliberately varied sample (mega-sites, SaaS, retail, news, a small niche tool):

| Field | Coverage |
|---|---|
| `domain`, `visits`, `authorityScore`, `globalRank`, `countryRank`, `bounceRate`, `pagesPerVisit`, `timeOnSiteSec` | 100% |
| `organicTraffic`, `paidTraffic`, `backlinks`, `referringDomains` + their change % | 100% |
| `topKeywords`, `trafficByCountry`, `aiTraffic`, `referralsIn`/`referralsOut`, all history series | 100% |
| `competitors` | ~85% - Semrush omits the set when too few organic keywords overlap |
| `paidChangePercent` | ~85% - no delta exists when the domain ran no paid search last month |
| `categoryRank`, `categoryRankName` | ~57% - only for domains Semrush has classified |
| `brandName`, `favicon` | ~57% - only where Semrush holds a brand entity |

**Not available, and we say so rather than shipping an empty column:** Site Audit metrics (site health, crawled pages, thematic reports) require a logged-in Semrush project - the public page ships them as zeros, so they are deliberately excluded from the record.

**Domains Semrush does not cover** (below its traffic threshold) end the run with a clear message naming them, and are **never charged**.

### Automate & schedule

Pair with Apify **Schedules** to refresh a competitor set weekly and push straight into Google Sheets, Slack or your warehouse via Apify integrations.

### Use with AI assistants (MCP)

Expose this actor as a tool to Claude, ChatGPT or any MCP client via the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp) and ask for competitive data in plain language.

### Agent-ready: autonomous payments (x402 & Skyfire)

This actor is **agent-ready**: AI agents can discover it, run it, and **pay for it autonomously**, with no Apify account and no human in the loop. It uses [pay-per-event](https://docs.apify.com/platform/actors/publishing/monetize/pay-per-event) pricing and [limited permissions](https://docs.apify.com/platform/actors/development/permissions), so it qualifies for Apify's agentic-payment standards:

- **[x402](https://docs.apify.com/platform/integrations/x402)**: an open, HTTP-native payment protocol. Agents pay per run in USDC on the Base network directly through the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp), with no account and no API key.
- **[Skyfire](https://docs.apify.com/platform/integrations/skyfire)**: agent-to-service payments for fully autonomous AI-agent workflows.

Building an AI agent, MCP tool, or autonomous data pipeline? This scraper is ready to plug in and pay as it goes.

### Pricing

Pay per event: **$0.006 per website overview**. Domains Semrush does not cover cost nothing.

### Integrate with any app

Connect the dataset to thousands of apps - no code required:

- **[Make](https://docs.apify.com/platform/integrations/make)** - multi-step automation scenarios.
- **[Zapier](https://docs.apify.com/platform/integrations/zapier)** - push new records straight into your CRM or spreadsheet.
- **[Slack](https://docs.apify.com/platform/integrations/slack)** - get notified when a scheduled run finds something new.
- **[Google Drive / Sheets](https://docs.apify.com/platform/integrations/drive)** - auto-export every run to a spreadsheet.
- **[Airbyte](https://docs.apify.com/platform/integrations/airbyte)** - pipe results into your data warehouse.
- **[GitHub](https://docs.apify.com/platform/integrations/github)** - trigger runs from commits or releases.

### More scrapers from scrapesage

Related Actors in the same category:

- **[Instagram Hashtag Scraper](https://apify.com/scrapesage/instagram-hashtag-scraper)** - Filtered Post Search
- **[Google Play Store Scraper](https://apify.com/scrapesage/google-play-scraper)** - Apps, Reviews & Developer Leads
- **[Keyword Research Scraper](https://apify.com/scrapesage/keyword-research-scraper)** - Google, YouTube, Amazon & More
- **[Google Trends Scraper](https://apify.com/scrapesage/google-trends-scraper)** - Interest, Regions & Trends
- **[Bing Ads Library Scraper](https://apify.com/scrapesage/bing-ads-library-scraper)** - Microsoft Ads Transparency

### FAQ

**Does this need a Semrush subscription?** No. It reads only the public overview pages Semrush publishes for search engines.

**Why is a domain "not covered"?** Semrush publishes an overview only above a traffic threshold. Use the registrable domain (`example.com`, not a sub-path), and expect very small or brand-new sites to be absent.

**Are the numbers Semrush's own?** Yes - they are Semrush's published estimates, read from the page as-is. No modelling or re-estimation.

**Can I import a big list?** Yes - paste it, link it, or use the Start-URLs *Link remote text file* option. A file that cannot be read says so and charges nothing.

### Disclaimer

**This Actor is an independent tool and is not affiliated with, endorsed by, or sponsored by Semrush Inc. or any of its subsidiaries. All trademarks mentioned are the property of their respective owners.**

"Semrush" and any related marks are the property of their respective owners and are used here only in a descriptive, nominative sense - to identify the publicly accessible website from which this Actor collects data. This Actor is not an official Semrush product, is not authorised or certified by Semrush Inc., and does not distribute Semrush software. It collects only publicly available information; you are responsible for ensuring your use of that data complies with applicable laws, regulations and the terms of the source website.

### Need help?

Open an issue on the actor's **Issues** tab - feature requests genuinely do get built.

***

*Not affiliated with, endorsed by, or sponsored by Semrush. "Semrush" is a trademark of its respective owner and is used here only to describe what this tool reads. Scrapes public pages only.*

# Actor input Schema

## `domains` (type: `array`):

Domains to analyse, one per line - e.g. <code>booking.com</code>, <code>stripe.com</code>, <code>bbc.co.uk</code>. Full URLs and Semrush overview links are accepted too and reduced to the registrable domain. Semrush only publishes an overview for domains above a traffic threshold; anything below it is reported as <i>not covered</i> and is never charged.

## `startUrls` (type: `array`):

Paste Semrush overview URLs directly, e.g. <code>https://www.semrush.com/website/booking.com/overview/</code>. <b>Bulk import:</b> use this field's <i>Link remote text file</i> option to load a hosted list (one entry per line) - or paste the list into <b>Import domains from a file</b> below.

## `urlsFromFile` (type: `string`):

Bulk-load domains without adding them one at a time. Two ways to use it: <b>(1)</b> paste your whole list straight in, one domain per line - open your .txt/.csv, select all, paste; or <b>(2)</b> give a single link to the file and it is downloaded for you - a public <code>.txt</code>/<code>.csv</code>, a Google Sheet or Drive share link (auto-converted to CSV export), or an Apify key-value-store record. Blank lines, <code>#</code> comments and extra CSV columns are ignored, and duplicates are only analysed once. If the file cannot be read the run says so and charges nothing.

## `includeHistory` (type: `boolean`):

Add the month-by-month series: visits, organic vs paid search traffic, backlinks and referring domains (up to 6 months each).

## `includeKeywords` (type: `boolean`):

Add the domain's top organic keywords with position, monthly search volume, CPC and share of traffic.

## `includeCompetitors` (type: `boolean`):

Add the competing domains Semrush ranks against this one, with a relevance score and their monthly visits.

## `includeCountries` (type: `boolean`):

Add the top countries sending traffic, each with its traffic share and desktop/mobile split.

## `includeAiTraffic` (type: `boolean`):

Add referral traffic coming from AI assistants (ChatGPT, Perplexity and friends) with volume and share - the newest signal on the page and the one most competitors do not export.

## `includeDevices` (type: `boolean`):

Add the desktop vs mobile visit counts and their month-by-month history.

## `includeReferrals` (type: `boolean`):

Add the sites sending traffic TO this domain and the sites it sends traffic to, with share and volume.

## `outputFields` (type: `array`):

Pick the fields you want and every record is trimmed to exactly those - handy for lean CSV/Sheets exports. <b>Nothing to memorise:</b> open the dropdown and tick, or start typing (<code>traffic</code>, <code>rank</code>, <code>backlink</code>) to filter. Leave empty to get every field.

## `maxItems` (type: `integer`):

Maximum number of domains to analyse in one run. Pagination stops automatically when this is reached.

## `proxyConfiguration` (type: `object`):

Proxy settings. Semrush serves these public pages cleanly to Apify datacenter proxies, which are used by default. If you ever see blocks, switch to Residential.

## Actor input object example

```json
{
  "domains": [
    "booking.com",
    "stripe.com"
  ],
  "urlsFromFile": "",
  "includeHistory": true,
  "includeKeywords": true,
  "includeCompetitors": true,
  "includeCountries": true,
  "includeAiTraffic": true,
  "includeDevices": true,
  "includeReferrals": true,
  "maxItems": 100,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

All analysed domains as JSON items in the default dataset.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "booking.com",
        "stripe.com"
    ],
    "urlsFromFile": ""
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapesage/semrush-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "domains": [
        "booking.com",
        "stripe.com",
    ],
    "urlsFromFile": "",
}

# Run the Actor and wait for it to finish
run = client.actor("scrapesage/semrush-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "booking.com",
    "stripe.com"
  ],
  "urlsFromFile": ""
}' |
apify call scrapesage/semrush-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapesage/semrush-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/aCSugNNd63ANKUgTq/builds/9YDmvt2M2cbhY3BXd/openapi.json