Cloudflare Just Gave Google a Deadline. Publishers Have 30 Days to Understand What It Means.

On September 15, Cloudflare changes its default settings for AI crawlers. The problem is that Googlebot is one of them. Here is what is happening, why it matters, and what publishers need to do before the deadline.

Key takeaways

• From September 15, Cloudflare will block “mixed-use crawlers” from any page that carries ads, by default
• Googlebot is classified as a mixed-use crawler because it indexes for search and trains Google’s AI models at the same time
• Publishers who block AI training bots risk accidentally disappearing from Google Search
• Cloudflare handles roughly 20% of global web traffic, which makes this enforceable at a scale no individual publisher could reach alone
• This is not a technical detail. It is a business decision that every publisher using Cloudflare needs to make in the next 30 days

Photo by Enchanted Tools on Unsplash

Most publishers have heard about Cloudflare in passing. It is the infrastructure company that sits between your website and the internet, handling traffic, security, and performance. Not the most exciting topic for a Monday morning.

But what Cloudflare announced on July 1 is something every publisher and advertiser should understand before September 15. Because it sets up a conflict that has been building for two years, and now has a hard deadline attached to it.

What Cloudflare actually announced

Before the implications, what exactly is changing.

Cloudflare processes roughly 20% of global web traffic. When you visit most websites, there is a good chance your request passes through Cloudflare’s infrastructure before reaching the site’s server. That position gives Cloudflare unusual visibility into who is crawling the web and why.

For the past year, Cloudflare has been categorizing web crawlers by purpose. A crawler can be classified as Search (it indexes pages for search engines), Training (it collects data to train AI models), or Agent (it acts on behalf of AI systems to complete tasks). Until now, publishers had to manually configure which crawlers to allow or block.

Starting September 15, that changes. The new default setting for new Cloudflare customers, new sites from existing customers, and all existing free-tier users will be:

Search crawlers: allowed. Training and Agent crawlers: blocked from any page that carries advertising.

There is also a new category that is the source of most of the confusion: mixed-use crawlers. These are bots that combine multiple purposes in a single crawler, doing search indexing, AI training, and agent tasks all at once. Under the new rules, mixed-use crawlers will be blocked from ad-supported pages by any configuration that blocks Training bots.

Cloudflare’s CEO Matthew Prince framed the rationale simply: “Most internet traffic is now non-human.” Automated bots now drive more than half of all web traffic, a milestone that arrived earlier than expected. The pitch to publishers is direct: stop giving your content away for free to systems that use it to avoid ever sending you a visitor.

The problem with Googlebot

Here is where it gets complicated for publishers.

Cloudflare has classified Googlebot as a mixed-use crawler. The reason: Google uses a single bot that simultaneously indexes pages for Google Search and collects data that feeds Gemini, AI Overviews, and AI Mode. When Googlebot crawls your page, it is doing both things at once.

This means that a publisher who uses Cloudflare’s Training block, including the legacy “Block AI bots” toggle, may also be blocking Googlebot. Not just the AI scraping. The search indexing too.

The implications are significant. If Googlebot cannot crawl your site, your pages stop appearing in Google Search results. For most publishers, Google still sends 30 to 60% of their traffic. Losing search visibility to protect content from AI scraping is, for many sites, a trade they cannot afford to make.

Google is aware of the problem. It already operates a separate crawler called Google-Extended, specifically for AI training purposes. That separation gives Google a compliance advantage: publishers who block Google-Extended can protect their content from AI training without affecting their search visibility. But the key word is “already.” Googlebot, the main crawler, still blends both functions.

Microsoft’s Bingbot and Apple’s Applebot face the same classification. Both are listed by Cloudflare as mixed-use crawlers that combine search and training in a single bot.

The numbers that explain why Cloudflare is doing this

Cloudflare published its own analysis of how much traffic different crawlers send back to publishers compared to how much content they take. The numbers are stark.

Google’s crawler generates roughly one referral visit for every 14 pages it crawls. That ratio has been declining as AI Overviews reduce the need for users to click through to publisher sites. OpenAI’s crawler ratio sits at approximately 1,700 pages crawled for every one referral visit. Anthropic’s crawler is at 73,000 pages crawled per referral visit.

In other words: AI companies are taking enormous amounts of publisher content and sending almost nothing back in return. Publishers bear the bandwidth cost. AI companies capture the value. Cloudflare’s move is explicitly designed to change that equation.

Get Alan Ronis’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The company is also expanding its Pay Per Crawl program, renamed to Pay Per Use. The original version charged AI companies when they fetched a page. The new model goes further: Cloudflare will pay publishers when their content shapes an AI answer, not just when it is fetched. Early partners are the AI search firms Ceramic.ai and You.com.

This is Cloudflare positioning itself as the infrastructure layer for what it calls Answer Engine Optimisation, the AI-era equivalent of SEO. Where SEO was about appearing in search results, AEO is about appearing in AI answers. Cloudflare wants to own the plumbing that makes that possible and profitable for publishers.

The impossible trade-off for publishers

The deeper problem is one that publishers have been sitting with for months, and the Cloudflare deadline forces it into the open.

Most publishers want two things at the same time. They want to appear in AI answers, because AI-referred traffic converts at significantly higher rates than traditional organic search traffic according to multiple studies. And they want to stop AI companies from scraping their content without compensation, because that scraping is partly responsible for the AI Overviews that are reducing their search traffic in the first place.

The problem is that protecting against scraping and appearing in AI answers often require the same crawler to have access. You cannot easily have both.

Cloudflare’s Pay Per Use program is designed to solve this: allow the crawl, get paid when your content is used. But the program is early, the payment rates are not published, and the AI companies that matter most (Google, OpenAI, Anthropic) are not yet committed partners.

For publishers with licensing leverage, the calculation is different. Reddit negotiated a data licensing deal with Google worth $60 million per year. The Associated Press has deals with OpenAI. These publishers can afford to block and negotiate from a position of strength.

For most publishers, that leverage does not exist. Blocking all AI crawlers protects content but risks invisible in both search and AI answers. Allowing all crawlers means continuing to fund AI systems with content that generates no meaningful return.

What publishers actually need to do before September 15

The good news is that the September 15 deadline applies to new Cloudflare customers, new sites, and free-tier users. Existing paid customers who have not changed their settings are not immediately affected.

But the broader decisions this forces are relevant for every publisher, regardless of their Cloudflare tier.

If you use Cloudflare, audit your bot management settings now. Understand what your current configuration is blocking and what it is allowing. The distinction between blocking Training bots and blocking mixed-use crawlers is subtle but consequential. A misconfigured setting can remove you from Google Search without you realising it.

If you do not use Cloudflare, the same logic applies through your robots.txt file and any other bot management tools you use. The question of which crawlers to allow is no longer a technical SEO detail. It is a business strategy decision that determines whether AI engines can cite your brand, recommend your content, and acknowledge you exist.

Understand what you are getting in return for your content. If Googlebot is crawling your site and you are appearing in Google Search, that is a return you can measure. If OpenAI’s crawler is taking your content and sending you 1 referral visit per 1,700 pages crawled, you should know that and decide whether the AI answer visibility you might be getting is worth it.

Watch how Google responds. If Google is serious about its search product, it has an incentive to fully separate Googlebot from its AI training crawler before September 15. That would resolve the publisher dilemma cleanly. If it does not, publishers face a harder choice.

My take

The Cloudflare announcement is the most concrete action taken by a major infrastructure company to address the AI scraping problem. And the scale matters: 20% of global web traffic is not a niche experiment. If Cloudflare holds the line on these defaults, AI companies will need to adapt their crawling behavior or lose access to a significant portion of the web.

What I find most interesting is the structural shift in who holds the power in this negotiation. For the past two years, the conversation about AI and publishers has been driven by what AI companies decide to do. OpenAI decides whether to license content or scrape it. Google decides whether Googlebot is a search crawler or an AI training bot. Publishers have been largely reactive.

Cloudflare shifts that. It sits between the content and the crawler. It can enforce rules that individual publishers cannot. And it is using that position to create a default that puts publishers in control of who accesses their content and on what terms.

Whether it works depends on whether AI companies comply, and whether publishers understand what choices they are actually making. The September 15 deadline is 30 days away. That is enough time to audit your settings, understand the trade-offs, and make a deliberate decision.

The publishers who treat this as a technical footnote are the ones who will wake up in October wondering why their traffic changed again.

Sources:

• Cloudflare gives AI crawlers a September deadline: pay publishers or get blocked — The Next Web, July 2, 2026
• Cloudflare will block mixed AI crawlers from ad-supported sites starting September 15 — Gagadget, July 3, 2026
• Cloudflare’s AI Bot Blocker Is Also Blocking Googlebot — Playwire, August 2026
• Cloudflare to block cynical search-and-scrape bots from ad-supported web pages — The Register, July 1, 2026
• Cloudflare’s new policy pushes AI companies to pay for publishers’ content — TechCrunch, July 1, 2026
• Cloudflare Sets September Deadline to Force AI Crawlers Apart From Search Bots — The AI Insider, July 2, 2026
• Cloudflare AI Crawler Rules: Will Your Block Hit Google? — SEO Kreativ, July 4, 2026
• OpenAI adds AppsFlyer attribution to ChatGPT ads as publishers begin blocking Google’s crawler — MarketScale, August 6, 2026