Cloudflare’s AI Crawler Bet: Can Matthew Prince Turn Blocking Into a Market That Pays Publishers?

On July 1, 2026, Cloudflare told its customers that Googlebot, Applebot and Bingbot would be blocked on any site that chose to block AI training, unless the owner opted out first. The rule was due to start on September 15. Search crawlers were about to be shut out alongside the training bots Cloudflare had spent a year fighting.

The plan changed. On September 15, Cloudflare released a setting called Disallow AI Training, which lets a site stay in search while refusing AI training, and said Apple, Google and Microsoft either honor it or have committed to honor it.

The gap between those two announcements is the clearest way into Cloudflare’s AI crawler policy. Matthew Prince, the company’s co-founder and CEO, began in July 2025 with a blunt default: AI crawlers pay or stay out. The problem has since proved harder than blocking bots. Some of the largest crawlers do two jobs at once, publishers depend on them for search traffic, and the payment system meant to replace lost visits is still in beta.

What was the old deal between Google and websites?

For almost thirty years the arrangement was easy to state. A site let Google copy its pages so they could appear in search, and Google sent visitors back. Those visitors could become advertising revenue, subscriptions or simply an audience.

Prince’s argument, set out in a Cloudflare post on July 1, 2025, was that the bargain had been weakening before generative AI arrived. He cited research that about three in four mobile queries ended without leaving Google, and wrote that earning the same traffic had become almost ten times harder in a decade. AI tools made it worse, he argued, putting OpenAI at 750 times harder and Anthropic at 30,000 times harder than the Google of old. Those are Cloudflare’s own estimates from a single date, so they are best read as the company’s case, not an industry standard.

Independent evidence points the same way on clicks, if not on scale. In a randomized field experiment, Saharsh Agarwal of the Indian School of Business and Ananya Sen of Carnegie Mellon found that outbound organic clicks fell 39.8% on searches where Google showed an AI Overview. It is a working paper, not yet peer reviewed, and it covers about 1,065 U.S. desktop Chrome users, so it does not describe every site or every search.

The commercial problem follows. Publishers still pay to report, write and host the content. If the answer reaches the reader without the visit, the deal that paid for that work stops working. On the second-quarter earnings call, Prince said more than half of the traffic crossing Cloudflare’s network was not human.

What did Cloudflare change in July 2025?

Cloudflare called it Content Independence Day. From July 1, 2025, every new customer domain would start with AI crawlers blocked unless they paid or the owner chose to allow them. Existing customers already had a one-click block. Alongside the default, Cloudflare opened a private beta of Pay Per Crawl, which lets a publisher charge an AI crawler for access using the long-dormant HTTP 402 “payment required” response, with Cloudflare acting as merchant of record.

Several publishers welcomed it. Roger Lynch, chief executive of Condé Nast, called it a critical step toward a fair value exchange, and Fortune and Quora backed the move in Cloudflare’s announcement.

Two points matter when judging the decision. First, the default was cheap to announce and hard to ignore. Prince said on the earnings call that more than 20% of the internet sits behind Cloudflare, so AI companies that want those pages have to account for the policy. Second, a default is not a market. Blocking is something Cloudflare can enforce on its own network. Payment needs a buyer willing to pay, a price both sides accept and a billing system both trust.

The customer overlap complicates things. On the same call, Prince said that over 80% of the major AI companies are Cloudflare customers. Being a customer is not the same as agreeing to pay publishers, and the policy only works if both sides stay.

Why did crawlers that do two jobs become the real problem?

Google was the hard case. Cloudflare’s argument centers on Googlebot, which crawls for Google Search, including AI features such as AI Overviews, as TechCrunch described it. As Cloudflare tells it, a publisher that blocks Googlebot to stop AI use also leaves search. At Bloomberg Tech in London in October 2025, Prince added that blocking Google’s crawler also blocks Google’s ad-safety checks, which would stop a publisher’s advertising across platforms.

Google’s answer at the time was that site owners can opt out of AI training with Google-Extended, which does not affect inclusion in Google Search. Google’s documentation describes that token as a control for Gemini training and for grounding in Gemini apps and Vertex AI. It is separate from Googlebot, and separate from Cloudflare’s own settings. Cloudflare’s filing to the UK regulator noted that AI features inside Search would need a different tool that Google had still to develop.

Cloudflare pressed the point with the UK Competition and Markets Authority. In a February 2026 filing, it called mandatory crawler separation the only viable technical solution, and said Googlebot saw about 1.8 times as many unique URLs as GPTBot and 1.7 times as many as ClaudeBot between December 2025 and January 2026.

The regulator took a different route. On June 3, 2026, the CMA imposed a publisher conduct requirement on Google. As the CMA describes it, Google must give publishers effective controls over how their search content is used in generative AI, explain that use clearly, provide engagement metrics and attribute content properly. The summary does not mention splitting the crawler.

In a September interview on The Verge’s Decoder podcast, as The Next Web reported, Prince said that after the UK ruling Google signalled its crawler would say what it is doing on each page, and that the change would apply beyond the UK. That is Prince’s account of Google’s intentions, not a Google statement. Nor does the record show that the ruling alone explains Cloudflare’s own policy changes.

How did Cloudflare’s Google plan change between July and September 2026?

The July 1 post introduced a sharper vocabulary. Cloudflare began sorting bots by behavior: Search, which builds an index and should send visitors back; Agent, which acts for a person in real time; and Training, which absorbs content into a model. It argued that bundled crawlers give large search companies an unfair advantage and push newer rivals toward evasive behavior.

Then came the line that mattered. From September 15, crawlers that combine search and training would be judged by all of their behaviors, and because the strictest rule wins, Googlebot, Applebot and Bingbot would be blocked for customers who chose to block training. Customers could opt out beforehand.

The September 15 post changed the design. Disallow AI Training is Cloudflare’s own setting, separate from Google-Extended. It publishes a no-training preference in robots.txt while accountable search crawlers keep indexing, and training-only crawlers from Amazon, Anthropic, Meta and OpenAI are blocked. Existing training blocks were migrated to the new setting. The stricter Block option remains, and it now does stop Googlebot, Applebot and Bingbot, search included. So Cloudflare does not block Googlebot by default; a site has to choose that.

Cloudflare’s own numbers show what was at stake: 17% of its sites use some mechanism to block training, while fewer than 1% block search bots. Those figures describe Cloudflare’s network, not the internet.

Cloudflare is open about the weakness of the tool it is publishing. A robots.txt file, its post says, cannot identify who is crawling, work out why, or stop a crawler that ignores it. Its answer is to identify and classify crawlers at the network, block those that ignore the preference, and report what each operator does on Cloudflare Radar.

The record does not say why Cloudflare moved from July’s wording to September’s, and guessing would add nothing. What it does show is that the new design leans on promises. Cloudflare says Apple, Google and Microsoft qualify as accountable through a mix of current tools and time-bound commitments. Microsoft is still building robots.txt support, targeted for early 2027, so Disallow AI Training does not yet reach Bing. Apple’s inspection tool is slated for next year. Cloudflare itself says AI summaries are the next problem.

What is Pay Per Use, and what has it proven?

On September 30, Cloudflare put Pay Per Use into beta. It moves the price from the fetch to the result. An AI company defines a specific use of content, such as returning an excerpt in a search answer, and offers a price. A publisher accepts or declines in a dashboard. The buyer reports each use, and Cloudflare bills the buyer and pays the publisher monthly.

Cloudflare’s logic is that AI products fetch far more than they use, so charging per crawl asks buyers to pay before they know what they need. Pay Per Crawl stays available for publishers who prefer it.

The company’s own post sets out the limits. Usage is self-reported: program terms require complete reporting, and Cloudflare checks that each reported use maps to an enrolled publisher, but the buyer, not Cloudflare, is the source of the usage record. Cloudflare says the beta exists to find out whether buyers and publishers both want to keep going. In July, it had described two early experiments, Ceramic.ai’s pay-per-query model and You.com’s on-demand payment for premium content, without publishing prices.

The September 30 announcement does not disclose prices, aggregate publisher earnings or completed payouts, and Cloudflare’s earnings call, as reviewed for this article, gives no such figure for either model. Second-quarter revenue of $696.1 million, up 36%, is company-wide, and the chief financial officer credited the Workers developer platform, agent workloads and large customers for the strength. It says nothing about whether this payment model works.

Where does Cloudflare’s own interest sit?

Cloudflare is not a neutral referee. It sells bot management and AI crawl controls, and with Pay Per Use it takes on billing and payouts. On the earnings call, Prince asked what would happen if being a Cloudflare customer meant receiving money, generated by micro-transactions largely from agents. That is a revenue ambition as much as a policy.

The customer overlap pulls the other way. Prince said Cloudflare blocks traffic that works against a customer’s business model and does not charge extra for it, yet the same network serves most of the major AI companies. Cloudflare’s commercial position depends on being useful to publishers and to the companies publishers are trying to charge.

Skeptics exist. Yahoo’s chief executive, Jim Lanzone, said recently that micropayments will never be enough for publishers, according to The Next Web. Prince has been wrong on timing before. He told The Verge that each of his earlier forecasts for bot traffic proved too slow, and he now expects automated traffic to reach about 1,000 times human traffic within five years.

What has to happen for the strategy to work?

The 2025 default showed that Cloudflare can change how a large part of the web treats AI crawlers. Whether that becomes a market depends on four conditions that remain unproven.

Buyers have to say yes. Pay Per Use only exists if AI companies make offers that publishers accept, at prices worth the effort. Nothing public yet shows either. Prince mentioned a research pilot with OpenAI on the earnings call, but described it as research and gave no terms.

The platforms have to deliver what they promised. Cloudflare’s accountable designation rests partly on commitments: URL-level reporting that Google told Cloudflare to expect within weeks, Microsoft’s robots.txt support in early 2027, and an Apple inspection tool due next year. If they arrive, a publisher can see which pages were offered for training and how they performed in search. If they slip, Disallow AI Training is a preference that is hard to check.

The plumbing has to scale. On the earnings call, Prince put the network at about half a billion requests per second and estimated, in figures he called unaudited, that perhaps 1% to 10% could be monetized through micro-transactions. That would mean 10 million to 100 million payments per second, against roughly 20,000 at Visa’s holiday peak, a number he gave from memory. These are Prince’s hypothetical estimates, taken from an automated transcript, not capacity Cloudflare has built. Whether Cloudflare builds that itself or finds partners, it is a different kind of engineering problem from blocking a bot.

Publishers have to see money. Pay Per Use and Pay Per Crawl will be judged less by design than by whether a publisher can point to a payout that offsets the visits it has lost. Cloudflare says publishers can track usage and estimated earnings in its dashboard, but estimated earnings are not verified payouts. Confirmation from publishers rather than from Cloudflare will matter.

If those conditions are met, the July 2025 default will look like the first move in a new market. If they are not, it will remain what it already is: a control that works, attached to a market that has not formed.