Ads.txt Scraper - Publisher Ad Seller Data

🔎 Ads.txt Scraper - Publisher Ad Seller Data

Run an IAB ads.txt audit on any list of publisher domains: get the authorized digital sellers, publisher IDs, DIRECT or RESELLER status and certification authority IDs, and optionally match every seller against the ad system's sellers.json for name, domain and type. Export to JSON/CSV/Excel, run on a schedule, call via API, or connect to Make, Zapier or n8n.

Data use. This Actor can return personal data (such as names, handles or contact details). You are the controller of the data you collect with it: make sure you have a lawful basis under the GDPR and similar laws, collect only what you need, and do not use it for unsolicited marketing.

What is the Ads.txt Scraper?

The Ads.txt Scraper turns public website standard files into clean structured data. These small files are easy to miss manually, but they are useful for audits, lead qualification, compliance checks, SEO operations, advertising research, and automated monitoring. Instead of opening every domain in a browser, you can provide a list of domains or direct file URLs and receive normalized rows in an Apify dataset.

The actor is intentionally lightweight. It uses direct HTTP, follows redirects, and parses the public text file without launching a browser. That keeps runs fast, inexpensive, and suitable for scheduled checks across many domains.

What data does it extract?

• Final source URL and input URL
• Domain and HTTP status code
• Content type
• Parsed fields relevant to this file type
• Raw line or raw text excerpts for auditability
• URLs and email addresses where present
• Structured result rows ready for export
• Source marker for downstream pipelines

Input

Field	Type	Description
targets	array	Domains or direct URLs to check. Domains are automatically converted to the standard file path.
maxResults	integer	Maximum rows to return per target. Useful for large files such as ads.txt or llms.txt.
resolveSellers	boolean	Default false. Looks up every row's publisher ID in the ad system's sellers.json and adds the seller's name, domain and type. No extra charge.

Example input

{
  "targets": [
    "https://www.cnn.com/ads.txt"
  ],
  "maxResults": 25
}

Seller names from sellers.json (optional)

Set resolveSellers to true to audit the supply chain: the actor fetches https://<ad system>/sellers.json once per ad system (cached for the run, 30 second timeout, files over 25 MB are skipped) and matches each row's publisher_id to seller_id. Each row gets sellerMatch (true, false, or null when the file could not be read), sellerName, sellerDomain and sellerType (PUBLISHER, INTERMEDIARY or BOTH). Ad systems that failed are listed with the error in the SELLERS_SUMMARY record of the run's key-value store. Rows cost the same as without it.

{"targets": ["example.com"], "maxResults": 25, "resolveSellers": true}

{"exchange_domain": "rubiconproject.com", "publisher_id": "1016", "relationship": "DIRECT", "sellerMatch": true, "sellerName": "The Meet Group Inc", "sellerDomain": "meetme.com", "sellerType": "PUBLISHER"}

Tutorial: check a publisher's authorized sellers in one run

Question this answers: who is allowed to sell ad space on a site, and is each seller the publisher itself (DIRECT) or a middleman (RESELLER)? ads.txt alone gives you account numbers. With resolveSellers on, the same run adds the name behind each number.

1. Start the run. Open the Input tab, add the domains and switch on "Resolve sellers.json". Or send this input through the API:

{"targets": ["nytimes.com", "cnn.com", "theguardian.com"], "maxResults": 300, "resolveSellers": true}

2. Read the table. We ran exactly this input on Apify (512 MB, platform usage $0.0021 for the run). Five rows from the dataset, columns trimmed:

domain	line	exchange_domain	publisher_id	relationship	sellerMatch	sellerName	sellerType
adstxt.dsc-ato.com	6	themediagrid.com	9UX1JV	DIRECT	true	Warner Media, LLC	publisher
adstxt.dsc-ato.com	12	pubmatic.com	161763	RESELLER	true	Saambaa LLC	INTERMEDIARY
www.theguardian.com	8	aps.amazon.com	3722	DIRECT	true	The Guardian	publisher
www.theguardian.com	33	pubmatic.com	160557	RESELLER	true	The Ozone Project	INTERMEDIARY
www.nytimes.com	25	adswizz.com	nytimes	DIRECT	false

Three things show up in these rows:

• cnn.com/ads.txt redirects to a hosted ads.txt manager, so domain shows adstxt.dsc-ato.com, the host that served the file. The input_url field keeps the domain you sent.
• A DIRECT line should resolve to the publisher or its parent company (The Guardian above). A RESELLER line resolves to a company that sells on the publisher's behalf (Saambaa, The Ozone Project). Check sellerType against relationship: a RESELLER line whose seller is typed PUBLISHER, or a DIRECT line whose seller is an unrelated intermediary, is worth a question to the publisher. sellerType keeps the casing the ad system wrote, so compare it without case.
• The domain nytimes.com has 30 lines in its file. 18 of them matched, 1 did not and 11 could not be checked.

3. What sellerMatch: false means. The ad system's sellers.json was read, and it does not list that publisher_id as a seller_id. For the adswizz.com row above, Adswizz's file has no seller nytimes. That is the signal to look at: an ID the ad system does not know can be a typo, a retired account or an unauthorized line. It is a lead for a manual check, not proof of fraud, because some ad systems publish a partial sellers.json or hide confidential sellers.

sellerMatch: null is different. It means the ad system's sellers.json could not be read, so no verdict is possible. In this run google.com answered HTTP 404 and amazon-adsystem.com dropped the connection, so every Google and Amazon line got null. The failed ad systems and their errors are in the SELLERS_SUMMARY record of the run's key-value store. Count null rows separately from false rows in your report.

In the first 300 lines of the CNN file (served from the host above): 205 matched, 13 did not, 82 could not be checked, and 138 of the matches were RESELLER lines.

Check weekly for new RESELLER lines

1. Save the input above as a Task (Actions, Save as a new task), with maxResults at 10000 so a long file is read to its end and the week-to-week comparison is not cut at an arbitrary line.
2. Open the Task's Schedules tab and add a weekly schedule, for example every Monday at 06:00.
3. Compare each run with the one before. A line counts as new when its domain, ad system, account ID and relationship are not in last week's dataset:

from apify_client import ApifyClient

client = ApifyClient("<your Apify token>")
last_two = client.task("<your task id>").runs().list(status="SUCCEEDED", desc=True, limit=2).items
key = lambda r: (r["domain"], r["exchange_domain"], r["publisher_id"], r["relationship"])
rows = lambda run: list(client.dataset(run["defaultDatasetId"]).iterate_items())
before = {key(r) for r in rows(last_two[1])}
for r in rows(last_two[0]):
    if r["relationship"] == "RESELLER" and key(r) not in before:
        print(r["domain"], r["exchange_domain"], r["publisher_id"], r.get("sellerName"), r["sellerMatch"])

To get this by email or in Slack, add a webhook on the Task's "run succeeded" event that calls a small script like this one. The same loop with before and the new rows swapped lists lines that disappeared.

For mobile apps, the same data sits in app-ads.txt files. The App-ads.txt Scraper takes an app-ads.txt file or an App Store or Google Play link and returns the same seller fields.

Output

{"exchange_domain": "google.com", "publisher_id": "pub-123456789", "relationship": "DIRECT", "certification_authority_id": "f08c47fec0942fa0", "source": "ads"}

Use cases

• SEO audits: collect machine-readable public metadata from client or competitor domains.
• Lead generation: qualify companies by their public operational, advertising, or contact signals.
• Compliance monitoring: schedule checks and detect missing, changed, or expired files.
• Advertising research: extract authorized seller relationships from publisher domains.
• AI visibility work: monitor llms.txt files and documentation links as AI search practices evolve.
• Data enrichment: add public standards metadata to domain lists, CRM accounts, and prospect databases.

Practical tips

For domain lists, pass bare domains such as example.com and the actor will request the standard path automatically. For custom locations, pass the full URL. Keep maxResults small for quick health checks and larger for full exports. When monitoring a portfolio, schedule the actor daily or weekly and compare rows by domain, source_url, and file-specific identifiers.

The actor returns raw excerpts because public standards files are not always perfectly formatted. Keeping the original line or text makes it easier to debug unusual publishers, malformed entries, or custom fields that are still valuable to a human analyst.

Data quality notes

Public website-standard files are simple on paper, but real-world files vary a lot. Some domains add comments, blank lines, custom fields, redirects, casing differences, or extra whitespace. This actor keeps both normalized fields and the original source values so you can trust the output in automated workflows while still having enough context for manual audits.

For files that contain multiple records, such as advertising seller lists or documentation indexes, the actor emits one row per useful record. For files that describe one website-level policy or team file, it emits one normalized row per target. That keeps the dataset natural to join with domain lists, CRM exports, spreadsheet audits, and BI tools.

Workflow ideas

Start with a list of customer domains, competitor domains, publishers, SaaS companies, app developers, or prospect websites. Run the actor once to create a baseline, then schedule it weekly to detect changes. You can send the output to a webhook, compare it with the previous dataset, and alert your team when a file appears, disappears, expires, changes seller relationships, or adds new contact details.

For agencies, this can become a repeatable audit product: upload a domain list, export the results, and include missing or malformed standards files in the client report. For lead-generation teams, the same data can help prioritize companies that show active advertising operations, security maturity, AI documentation readiness, or public site-maintenance signals.

Reliability and performance

This actor does not use a browser, paid unblocker, or residential proxy. It is designed for public text files that are meant to be fetched by automated systems. If a target returns no file, the actor logs the failure and continues with the next target. That makes it safe for larger domain lists where some websites are expected to be missing the standard file.

Because the parser is conservative, it avoids pretending malformed files are perfect. Rows include HTTP metadata and raw source values so your workflow can decide whether to accept, clean, or review them.

FAQ

Does it require an account? No. It reads public files from the provided websites.

Can I pass domains instead of full URLs? Yes. Bare domains are converted to the correct standard path.

Can I run it on many websites? Yes. Add multiple targets and use Apify scheduling for recurring checks.

Does it crawl the whole website? No. It fetches only the standard file URL for each target.

Can I export to Excel? Yes. Apify datasets export to JSON, CSV, Excel, XML, and RSS.

Can I connect it to automation tools? Yes. Use Apify API, webhooks, Make, Zapier, n8n, or dataset exports.

What happens when a file is missing? The actor logs the target and continues with the next one.

Is it legal? The actor reads public website files. You are responsible for respecting website terms, rate limits, privacy rules, and applicable laws.

How is billing calculated? Pay-per-event billing charges per result pushed to the dataset.

Why include raw text? Raw excerpts make audits easier and preserve fields that may not be part of the common format yet.

Related tools

• App-ads.txt Scraper: check the mobile side of the same publisher, from an app-ads.txt file or an App Store or Google Play link
• App Store Reviews Scraper: use it when the publisher also runs iOS apps and you want to read their reviews
• Website Contact Extractor
• Meta Tags Extractor
• HTTP Headers Checker
• Robots Sitemap Analyzer
• Schema Markup Extractor

Keywords: ads.txt scraper, ads.txt audit, IAB ads.txt, sellers.json, authorized digital sellers, publisher ad sellers, programmatic ads data, ad verification scraper, web standards scraper, domain audit tool, website metadata extractor, Apify scraper, SEO automation, lead generation data, compliance monitoring, website intelligence