AI Crawler Access Checker

Find out which AI bots a website lets in, for up to 5,000 sites per run. The checker reads each site's robots.txt the way the bots do and reports, for 16 AI crawlers such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended, whether they may fetch the home page and whether the site has a rule just for that bot. It also checks llms.txt, llms-full.txt and the X-Robots-Tag header. For SEO and GEO specialists, publishers, agencies and developers who need a quick, repeatable AI crawler check.

What is AI Crawler Access Checker?

It is a bulk AI crawler checker and robots.txt checker for AI bots. More and more content reaches people through AI assistants and AI search, and whether a bot can read your pages is decided by a few lines in robots.txt. Those lines are easy to get wrong: a rule written for one bot does not apply to another, a wildcard group can block more than intended, and a forgotten Disallow: / can keep every AI search engine away. This Actor shows the effective result per bot, so you do not have to read and interpret the file yourself.

For each site you enter, it:

1. reads robots.txt once, with the matching rules of RFC 9309 (longest matching path wins, Allow wins a tie, * and $ in patterns are understood),
2. works out, for each of 16 known AI bots, whether it may fetch the home page,
3. checks whether the file has a group for the bot's own name, or only the general * group,
4. looks for /llms.txt and /llms-full.txt and reports whether they exist and how large they are,
5. reads the response headers of the home page for noai, noimageai, noindex and nofollow in X-Robots-Tag.

It is a read-only check of public files. It never returns the text of robots.txt, llms.txt or any page, only flags and counts.

What does it check?

Per AI bot (16 bots):

Bot	Operator	Purpose
GPTBot	OpenAI	training
OAI-SearchBot	OpenAI	search
ChatGPT-User	OpenAI	user request
ClaudeBot	Anthropic	training
Claude-SearchBot	Anthropic	search
Claude-User	Anthropic	user request
Google-Extended	Google	training
Applebot-Extended	Apple	training
PerplexityBot	Perplexity	search
Perplexity-User	Perplexity	user request
Meta-ExternalAgent	Meta	training
CCBot	Common Crawl	dataset
Bytespider	ByteDance	training
Amazonbot	Amazon	search
DuckAssistBot	DuckDuckGo	search
cohere-ai	Cohere	training

For every bot you get allowed (can it fetch the home page?) and own_rule (does robots.txt contain a group for this bot's name?). The purpose labels are the Actor's own grouping of the bots: training and dataset bots collect content for model training or datasets, search bots build an AI search or answer index, and user request bots fetch a page when a person asks an assistant to.

Per site:

• Counts and lists: how many bots are allowed and blocked, which training and dataset bots are blocked, and which AI search bots are allowed.
• robots.txt status: parsed, missing (the site answered with a 4xx status, so everything is allowed) or unreachable (a server error, timeout or connection problem, so the answer for each bot is unknown), plus the number of Sitemap: lines it lists.
• llms.txt and llms-full.txt: present or not, size in bytes, number of lines, and whether the file was longer than the 1 MiB the Actor reads.
• X-Robots-Tag: which of noai, noimageai, noindex, nofollow appear in the header of the home page.

Use cases

• Protect your content from AI training: confirm that GPTBot, ClaudeBot, Google-Extended, CCBot and the other training bots are really blocked after you edited robots.txt.
• Get found in AI search: check that OAI-SearchBot, Claude-SearchBot, PerplexityBot and the other search bots are allowed, for your own site or for a client's.
• GEO and AI visibility audits: add an AI crawler section to every site audit and see in one table which bots a site welcomes.
• Market research: check hundreds or thousands of publishers, shops or competitors in a market and count how many block training bots but allow search bots.
• Find accidents: a staging robots.txt with Disallow: / that went live, or a wildcard group that shuts out every bot, shows up as 16 blocked bots.
• llms.txt rollout: find which of your sites already publish llms.txt and llms-full.txt and which are missing one.
• Monitoring: schedule a weekly run on your domains and compare the results to catch a robots.txt change that nobody announced.
• Agency and compliance reporting: attach a dataset of per-bot decisions to a client report as evidence of the policy the site publishes.

How to use AI Crawler Access Checker

1. Open the Actor in Apify Console and click Try for free or Start.
2. Enter your sites in Sites, one per line (example.com or https://www.example.com/).
3. Leave Timeout (seconds) at 15, or raise it for slow servers.
4. Set a maximum cost per run in the run options if you want a hard spending limit; the Actor then checks only as many sites as the limit pays for.
5. Click Start. The status message shows the number of sites being checked.
6. Open the Output tab and download the results as JSON, CSV, Excel, XML or HTML. The Overview view shows the main columns at a glance.
7. To monitor, create a Schedule (for example weekly) with the same input.

Input

Field	Type	Default	What it does
sites	array of strings	required	The websites to check. Accepts example.com, https://www.example.com/ or objects like {"url": "https://example.com"}. At most 5,000 different sites per run; exact duplicates are removed.
timeoutSecs	integer	15	How long to wait for each response, from 5 to 60 seconds.

Tips:

• Only the origin matters. A path in your input is ignored: https://example.com/blog/post checks https://example.com/. The scheme and port you give are kept, and https:// is assumed when you leave the scheme out.
• www.example.com and example.com are different hosts with their own robots.txt. Check the one your visitors and bots actually land on, or both.
• Bots are matched on the home page. A site can block bots only from some sections, for example /private/; the result then still says allowed: true for the home page.

{
  "sites": ["https://www.example.com/", "example.org", "news.example.net"],
  "timeoutSecs": 15
}

Output

One dataset item per input site, in input order. Example:

{
  "input": "https://www.example.com/",
  "site": "https://www.example.com/",
  "robots_txt": {"status": "parsed", "sitemaps": 1, "error": ""},
  "agents": [
    {"agent": "GPTBot", "operator": "OpenAI", "purpose": "training", "allowed": false, "own_rule": true},
    {"agent": "OAI-SearchBot", "operator": "OpenAI", "purpose": "search", "allowed": true, "own_rule": false},
    {"agent": "ChatGPT-User", "operator": "OpenAI", "purpose": "user request", "allowed": true, "own_rule": false},
    {"agent": "ClaudeBot", "operator": "Anthropic", "purpose": "training", "allowed": true, "own_rule": false},
    {"agent": "Claude-SearchBot", "operator": "Anthropic", "purpose": "search", "allowed": true, "own_rule": false},
    {"agent": "Claude-User", "operator": "Anthropic", "purpose": "user request", "allowed": true, "own_rule": false},
    {"agent": "Google-Extended", "operator": "Google", "purpose": "training", "allowed": false, "own_rule": true},
    {"agent": "Applebot-Extended", "operator": "Apple", "purpose": "training", "allowed": true, "own_rule": false},
    {"agent": "PerplexityBot", "operator": "Perplexity", "purpose": "search", "allowed": true, "own_rule": false},
    {"agent": "Perplexity-User", "operator": "Perplexity", "purpose": "user request", "allowed": true, "own_rule": false},
    {"agent": "Meta-ExternalAgent", "operator": "Meta", "purpose": "training", "allowed": true, "own_rule": false},
    {"agent": "CCBot", "operator": "Common Crawl", "purpose": "dataset", "allowed": false, "own_rule": true},
    {"agent": "Bytespider", "operator": "ByteDance", "purpose": "training", "allowed": false, "own_rule": true},
    {"agent": "Amazonbot", "operator": "Amazon", "purpose": "search", "allowed": true, "own_rule": false},
    {"agent": "DuckAssistBot", "operator": "DuckDuckGo", "purpose": "search", "allowed": true, "own_rule": false},
    {"agent": "cohere-ai", "operator": "Cohere", "purpose": "training", "allowed": true, "own_rule": false}
  ],
  "allowed_count": 12,
  "blocked_count": 4,
  "training_blocked": ["Bytespider", "CCBot", "GPTBot", "Google-Extended"],
  "search_allowed": ["Amazonbot", "Claude-SearchBot", "DuckAssistBot", "OAI-SearchBot", "PerplexityBot"],
  "llms_txt": {"present": true, "bytes": 2140, "lines": 48, "truncated": false, "reason": ""},
  "llms_full_txt": {"present": false, "bytes": null, "lines": null, "truncated": null, "reason": ""},
  "x_robots_tag": [],
  "checked_at": "2026-10-20T08:15:42Z",
  "error": ""
}

An input that is not a public site is reported like this, without the other fields:

{"input": "exmaple.invalid", "site": null, "error": "dns_error"}

Output fields

Field	Type	Meaning
input	string	Your input, trimmed and cut at 500 characters.
site	string or null	The home page that was checked (origin/). null when the input is not a public site.
robots_txt.status	string	parsed, missing (4xx: everything allowed) or unreachable (the file could not be read).
robots_txt.sitemaps	integer	Number of http or https Sitemap: lines in robots.txt.
robots_txt.error	string	Why robots.txt was unreachable, for example timeout, tls_error, connection_error or robots_unavailable (a non-2xx, non-4xx answer). Empty otherwise.
agents	array	One entry per bot, in the order of the table above.
agents[].agent	string	The bot's product token as used in robots.txt.
agents[].operator	string	The company behind the bot.
agents[].purpose	string	training, search, user request or dataset.
agents[].allowed	boolean or null	May the bot fetch the home page? true when robots.txt is missing, null (unknown) when it was unreachable.
agents[].own_rule	boolean	true when robots.txt has a group that names this bot, not only the * group.
allowed_count	integer	Bots with allowed: true.
blocked_count	integer	Bots with allowed: false. Bots with null count in neither.
training_blocked	array of strings	Blocked bots with purpose training or dataset, sorted.
search_allowed	array of strings	Allowed bots with purpose search, sorted.
llms_txt	object	The /llms.txt check: present (true, false, or null when it could not be looked at), bytes, lines, truncated, reason.
llms_full_txt	object	The same for /llms-full.txt.
x_robots_tag	array or null	Which of noai, noimageai, noindex, nofollow the home page's X-Robots-Tag contains; empty when none, null when the home page could not be requested.
checked_at	string	UTC time of the check.
error	string	Empty, or the reason the site could not be checked: invalid_url, dns_error, not_public_address, internal: <ErrorName>, or the robots.txt problem when it was unreachable.

In the file objects, present is false when the server answers with anything other than 200, with an empty body, or with an HTML page (a catch-all "not found" page is not an llms.txt). When present is null, reason says why the file was not looked at: the site's robots.txt forbids the request (blocked_by_robots), or the request failed (timeout and similar).

Dataset view

The Overview view lists site, allowed and blocked counts, blocked training bots, allowed search bots, whether llms.txt is present and any error. All fields are available in every export format.

How much does it cost?

This Actor uses pay-per-event pricing: you pay USD 0.003 per site checked (USD 3.00 per 1,000 sites), plus Apify's standard actor start event of USD 0.00005 per run. With pay-per-event pricing the platform usage of the run is included in these prices.

Sites checked	Site events	Actor start	Total
100	USD 0.30	USD 0.00005	USD 0.30005
1,000	USD 3.00	USD 0.00005	USD 3.00005
5,000 (the maximum per run)	USD 15.00	USD 0.00005	USD 15.00005
10,000 (2 runs of 5,000)	USD 30.00	2 x USD 0.00005	USD 30.0001
100,000 (20 runs of 5,000)	USD 300.00	20 x USD 0.00005	USD 300.001

What is charged and what is free:

• Charged: each site for which at least one request was made, which includes sites whose robots.txt was unreachable.
• Free: inputs that are not a public site (invalid URL, a host that does not resolve, a host that resolves to a private address), and the rare site that fails with an internal error. These items still appear in the dataset with their error.

Set a maximum cost per run and the Actor checks only as many sites as the limit allows. It cuts the list from the end, so put the sites you care about most first. Apify's free plan includes monthly platform credit, which you can use to try the Actor.

Tips and best practices

• Read own_rule together with allowed. A bot with allowed: false and own_rule: true was blocked on purpose. A bot with allowed: false and own_rule: false is caught by the * group, which usually means the whole site is closed to every crawler.
• Check both hosts. If your site answers on example.com and www.example.com, the redirect hides that each has its own robots.txt; the Actor requests the host you give it.
• Do not confuse blocking with opting out of search. Google-Extended and Applebot-Extended are control tokens for AI training; they do not change how normal search crawlers treat your site. The Actor reports what robots.txt says, not what a search ranking does with it.
• Put important sites first if you use a spending limit.
• Use a schedule to catch changes, and compare blocked_count or training_blocked between runs, for example with a webhook that posts to your chat.
• Combine with the other tools: the Website Security Headers & TLS Audit covers the technical hygiene of the same domains, and the Sitemap URL Extractor lists the pages the bots would find.

Limitations

• robots.txt is a request, not a lock. It tells well-behaved bots what to do. The report shows what a site asks for, not what every bot actually does. Blocking at the server or firewall level is invisible to this check.
• Home page only. The decision is for the home page of the origin. Rules for other paths are not evaluated.
• The bot list is fixed. It covers the 16 bots in the table. A bot that is not listed, or that changes its name, is not reported, and a site's own rules for it cannot be seen here.
• Rules are matched by product token. A User-agent line counts for a bot when it names the bot's token (case does not matter, a trailing version such as /1.0 is ignored); otherwise the * group applies. Groups that name the same bot are combined.
• Unreachable robots.txt means unknown. For a server error, a timeout or a connection problem, allowed is null; the Actor does not guess.
• Redirects. robots.txt that redirects to another host is followed and applies, as the standard says. Up to 10 redirects are followed.
• Size limits. At most 500 KiB of robots.txt is read, and at most 1 MiB of llms.txt or llms-full.txt (truncated tells you when a file was longer). The text itself is never returned.
• Polite pace. There is at least half a second between two requests to the same host. A crawl delay in robots.txt slows the Actor further, up to 30 seconds. Different hosts are checked at the same time.
• Public sites only. Hosts that resolve to private, loopback or other non-public addresses are refused. URLs with a user name or password are not checked.
• Our own requests respect robots.txt. If a site's robots.txt disallows our user agent for /llms.txt or for the home page, those files are not requested and the result says so in reason.

FAQ

Is it legal to use? It reads files that a site publishes for crawlers (robots.txt, llms.txt) and the response headers of the home page, and it never stores or returns page text. Check only sites you own or may check, and see the fair use rules at the end of this page.

Does it respect robots.txt? Yes. It reads robots.txt to answer your question and obeys it for its own requests as well. It identifies itself with the user agent HelveticLabs-WebCheck.

Does it store personal data? No. The output contains flags, counts and URLs of the home page. It does not read or return e-mail addresses, cookies or page content.

How fast is it? Up to ten requests run at the same time, one per host at a time with a pause of at least half a second. Each site needs a few requests (robots.txt, llms.txt, llms-full.txt and one header request for the home page), so thousands of sites take a while but not long. Slow servers are limited by timeoutSecs.

Why is allowed true for every bot on some sites? Either robots.txt does not exist (status missing), or it has no rule that matches the bot. The default of the standard is that crawling is allowed.

Why is allowed null? robots.txt could not be read (status unreachable), so the decision is unknown. Check robots_txt.error for the reason and try again later.

Why was a site not charged? Inputs that are not public sites are free: invalid URLs, hosts that do not exist and hosts on private addresses. Everything else, including sites that fail afterwards, is charged because requests were made.

What do I do with llms.txt? llms.txt is a proposed file where a site describes itself for AI systems. The Actor only reports whether it exists, its size and number of lines, so you can see who publishes one. It does not judge the content and not whether any AI reads it.

Can I schedule it? Yes, with a schedule in Apify Console, for example weekly. Combine it with a webhook to be told when a run finishes.

Can I use it from Python, JavaScript or the API? Yes, see Integrations below.

Can AI agents use it? Yes, through Apify's MCP server. An agent can ask "which AI bots does this site allow?" and get the per-bot table back.

What happens on errors and timeouts? A site that fails is reported with its error and the run goes on. Results are pushed in batches, so results already written stay in the dataset when a run ends early.

Integrations

The actor ID is HelveticLabs/ai-crawler-checker.

Python (pip install apify-client):

from decimal import Decimal

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("HelveticLabs/ai-crawler-checker").call(
    run_input={"sites": ["example.com", "example.org"], "timeoutSecs": 15},
    max_total_charge_usd=Decimal("1"),  # optional spending limit for this run
)
# apify-client 3.x; with older versions use run["defaultDatasetId"]
for item in client.dataset(run.default_dataset_id).iterate_items():
    if item["site"] is None:  # not a public site: only "input" and "error"
        print(item["input"], item["error"])
    else:
        print(item["site"], item["allowed_count"], "allowed,", item["blocked_count"], "blocked")

JavaScript (npm install apify-client):

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('HelveticLabs/ai-crawler-checker').call(
    { sites: ['example.com', 'example.org'], timeoutSecs: 15 },
    { maxTotalChargeUsd: 1 },
);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.filter((i) => i.site).map((i) => [i.site, i.training_blocked]));

HTTP API: POST https://api.apify.com/v2/acts/HelveticLabs~ai-crawler-checker/runs with the input as the JSON body; add maxTotalChargeUsd as a query parameter to cap the cost.

No-code and automation: connect the Actor to Make, Zapier or n8n, or add a webhook that fires when a run finishes, for example to post sites with blocked search bots to a chat.

AI agents (MCP): Apify's MCP server lets AI agents and MCP clients run this Actor as a tool; pass maxTotalChargeUsd in the call options to cap the spend.

Related HelveticLabs actors

Actor	What it does
URL Status & Redirect Checker	Status code, redirect chain, final URL and response time for up to 100,000 URLs.
Website Health Audit: Broken Links & SEO Checker	Crawls a site for broken links and on-page SEO issues.
Website Security Headers & TLS Audit	A+ to F security grade: HTTPS, headers, SSL certificate, SPF and DMARC.
Sitemap URL Extractor & Validator	Every URL a site lists in its sitemaps, with validation.

Support and feedback

A bot that is missing, a result that looks wrong, or an idea? Open an issue in the Issues tab on this Actor's page, or write to contact@helveticlabs.org. Include the run ID and the site if you can.

Provider, privacy and fair use

Provider: HelveticLabs, sole proprietorship of Marvin Krieg, c/o SwissMailBox 170, 12 Rue Le Corbusier, 1208 Genève, Switzerland. E-mail: contact@helveticlabs.org. Imprint: https://www.helveticlabs.org/imprint

Privacy: This Actor runs on the Apify platform, in your Apify account. It processes only the URLs or domains you enter and the public technical data the checked servers return (status codes, headers, certificates, DNS records, sitemap URLs, links). It does not collect personal data: it never stores or returns page text (where a page's HTML is read, it is read in memory only, to find links and measure tags), and it leaves out e-mail addresses, cookie names and values and similar values. Your input and results stay in your Apify account and are stored by Apify under its own terms (https://apify.com/privacy-policy). The provider sees only the usage statistics Apify shows to Actor authors (such as the number of users and runs), not your input or results, unless you share a run for support. Questions about privacy: contact@helveticlabs.org.

Fair use: Check only sites you own or may check. Do not use the results to attack systems or people. The Actor respects robots.txt, identifies itself, and keeps a polite pace per site.

No warranty: The results are automated checks and can be incomplete. They are not a penetration test, legal advice or a compliance certificate.