Cloudflare, a company best known for its web infrastructure services, is stepping into the regulatory fray over artificial intelligence, calling for sweeping changes to how tech giant Google handles the web’s most valuable resource: its data. On October 21, 2025, Cloudflare’s CEO Matthew Prince took the stage at the Bloomberg Tech conference in London, making a direct appeal to the UK’s Competition and Markets Authority (CMA). His message? The time has come to force Google to separate its search crawler from the systems that feed its voracious AI engines.
This isn’t just a technical squabble over how bots index websites. At stake, according to Prince and a growing chorus of media executives, is the future of competition—and even the open web itself. For years, Google has used its ubiquitous web crawler, Googlebot, to index sites for search. But as the company pivots toward AI-powered products, that same crawler is now gathering content for training large language models and powering features like AI Overviews. Prince argues this dual use gives Google a massive, and arguably unfair, advantage over rivals like OpenAI, Anthropic, and Perplexity, who must identify their bots and often pay for access to content.
“Google is saying, ‘We have an absolute God-given right to all of the content in the world, even if we don’t pay for it, because look what we did for the last 27 years,’” Prince told the audience, according to Bloomberg. “And they’re saying we can take it and use the same crawler we use for search in order to power our AI systems. And if you want to opt out of one, you have to opt out of both.”
The CMA, which earlier this month officially designated Google as having a “substantial and entrenched” position in the search and advertising markets, now has the authority to impose stricter regulations. Prince’s proposal is straightforward: require Google to run its AI data-harvesting under a separate user agent, with distinct controls. That way, website owners could refuse to have their content used for AI training without losing their search visibility or breaking Google’s ad delivery, a dilemma many publishers currently face.
The consequences of the current arrangement are stark. Blocking Google’s crawler doesn’t just mean disappearing from search results—an existential risk for publishers who rely on search traffic for up to 20% of their revenue, according to industry surveys. It also disrupts Google’s ad safety and measurement systems, which can cripple advertising across all platforms. “But it gets even worse. If you block Google’s crawler, it blocks their ad safety team, which means that your advertisements across all of your platforms stop working, which is just a non-starter,” Prince explained, as reported by TechCrunch.
Cloudflare, which now runs a marketplace allowing websites to charge AI bots for scraping their content, says it’s uniquely positioned to weigh in. “We don’t have a dog directly in the fight. We’re not an AI company,” Prince said. “We’re not a media publisher, but we’re this network that sits between them—80% of the AI companies are our customers.” That vantage point has provided Cloudflare with empirical evidence about how Google’s crawler works and why it’s so difficult for others to replicate its reach and influence. The company has already shared this data with the CMA, hoping to inform the regulator’s next steps.
Cloudflare’s stance is winning allies among publishers and media executives, many of whom have long felt strong-armed by Google’s bundled approach to crawling. Neil Vogel, CEO of People, Inc.—the largest digital and print publisher in the United States—has been especially vocal. He called Google a “bad actor” and echoed Prince’s concerns, telling the Financial Times that media companies have no real choice but to let Google crawl their sites for AI content. Vogel’s company has adopted Cloudflare’s solution to block AI crawlers that don’t pay, and he reported that deal discussions are now underway with several large language model providers.
The issue is particularly acute in the UK, where Google’s share of the search market consistently tops 90%, according to StatCounter. Under the UK’s new digital markets regime, the CMA’s Digital Markets Unit has broad powers to designate companies with strategic market status and apply rules to prevent them from abusing dominance in one market to gain leverage in another. For Prince and his supporters, unbundling Google’s crawlers is a classic interoperability fix: separate identities, transparent controls, and no penalties for sites that opt out of AI ingestion.
“You could lose all Google visibility overnight and make no revenue—a lever no AI startup can pull,” Prince said, framing the dilemma faced by publishers. For news organizations and commerce sites, search is a lifeline, and the threat of losing it is simply too great to risk blocking Google’s crawler, even if it means their content is used for purposes they never agreed to.
What would unbundling look like in practice? Experts suggest it would require more than just a new user agent string. The solution would involve a package of measures: granular opt-ins for different AI capabilities, clear explanations of how data are used, robust audit trails, and fair, standardized licensing pathways. Crucially, there would be no search ranking penalties, ad stack disruptions, or feature downgrades for sites that refuse AI ingestion. Reporting on crawl volumes and model inputs would allow regulators and rights holders to monitor compliance, borrowing tools from classic competition enforcement but updating them for the quirks of generative AI.
Google, for its part, says it provides publisher controls and honors robots directives, and that its search and AI features deliver value to consumers and drive traffic back to the open web. The company is also rolling out new labels and settings associated with AI summaries. The question for the CMA is whether these measures are clear, effective, and independent enough to prevent Google’s search dominance from bleeding into AI.
Neil Vogel, CEO of Dotdash Meredith, has expressed similar concerns, telling the Financial Times that his company is deploying tools to block non-paying bots while negotiating licenses with big model providers. He, too, sees the current system as an unsolvable dilemma for publishers who want to maintain both search visibility and control over their data.
For Cloudflare, the stakes go beyond business interests. Prince’s proposal is about preserving the open web and fostering a functioning market for data—one where thousands of websites can license content to thousands of AI companies, without being forced into all-or-nothing choices. As he put it, “The solution is to foster a lot of competition in the market, where potentially thousands of AI companies could compete to buy content from thousands of media businesses and millions of small businesses.”
As regulators in the UK and beyond weigh their next moves, the outcome of this debate could shape the very fabric of the digital world. For now, the battle lines are drawn, and the future of web crawling—and the open web—hangs in the balance.
