robots.txt, llms.txt & security.txt Checker (Bulk)

robots.txt, llms.txt & security.txt Checker – bulk well-known files audit

Check the well-known files of many websites in one run: robots.txt, llms.txt, llms-full.txt, security.txt (RFC 9116), ads.txt, app-ads.txt and humans.txt. Each file is fetched, validated (soft-404 HTML pages are detected) and parsed into clean JSON.

What you get per site

• robots.txt: user-agent groups, declared Sitemaps, crawl-delay, whether all bots are blocked, and a per-bot verdict for 24 AI crawlers (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, CCBot, Google-Extended, PerplexityBot, Bytespider, Amazonbot, Applebot-Extended, meta-externalagent, cohere-ai…) plus Googlebot/Bingbot: allowed, partial, blocked or blocked (via *).
• llms.txt / llms-full.txt: present?, H1 title, summary, sections, link count.
• security.txt: Contact, Expires (expired? days left), Encryption, Policy, Hiring, Canonical, PGP-signed, RFC 9116 issues. Checks /.well-known/security.txt then /security.txt.
• ads.txt / app-ads.txt: DIRECT vs RESELLER records, number of ad systems, OWNERDOMAIN/MANAGERDOMAIN.
• humans.txt: present?
• A short issues list (no robots.txt, no sitemap declared, no security.txt, expired security.txt…).

Use cases

• See which sites block AI crawlers (GPTBot, ClaudeBot, CCBot…) across a list of publishers or competitors
• llms.txt adoption research / GEO (generative engine optimization) audits
• Security & bug-bounty recon: find security.txt contacts and VDP policies at scale
• SEO audits: sitemap discovery, crawl rules, ads.txt sanity checks

Input

{ "sites": ["apify.com", "github.com", "nytimes.com"], "includeContent": false }

Optional: files (subset of robots, llms, llmsFull, security, ads, appAds, humans), includeContent (raw text), timeoutSecs, concurrency, maxSites.

Output (one item per site)

{
  "site": "https://github.com",
  "hasRobotsTxt": true,
  "hasLlmsTxt": true,
  "hasSecurityTxt": true,
  "aiCrawlersBlocked": ["Bytespider"],
  "sitemaps": [],
  "securityContact": ["https://hackerone.com/github"],
  "issues": [],
  "files": {
    "robots": { "found": true, "statusCode": 200, "aiCrawlers": { "GPTBot": "allowed", "Bytespider": "blocked" }, "sitemaps": [] },
    "security": { "found": true, "contact": ["https://hackerone.com/github"], "expires": "2027-…", "expired": false },
    "llms": { "found": true, "title": "GitHub", "linkCount": 40 }
  }
}

Pricing

$0.0003 per site checked ($0.30 per 1,000 sites, all files included) + a tiny start fee. Unreachable sites are free.

Notes

Only public, well-known files are requested (one request per file, polite user agent). No login, no crawling.

Related Actors

• HTTP Security Headers Checker
• Sitemap URL Extractor
• Website Metadata Extractor
• SEO Audit Crawler