Cloudflare announced that, beginning September 15, 2026, its default settings will block “mixed-use” web crawlers — those that combine traditional search crawling, agent use, and AI training — from crawling pages that host ads. The company said the new defaults will apply to new Cloudflare customers, new sites created by existing customers, and all existing free customers, unless site owners change their settings.
Practical effect
Under the new default, crawlers that mix search, agent activities and training will be prevented by default from accessing pages containing advertisements. Site owners can override the default if they wish to permit access. Cloudflare warned that the change could affect how AI model providers obtain web content for training and for powering agentic services.
Why Cloudflare is making the change
Cloudflare says most website owners want their content to be discoverable in search and often via AI services, but also want protection against having their intellectual property used freely for AI training. In his announcement, Cloudflare co-founder and CEO Matthew Prince pointed to a recent milestone where bot traffic surpassed human traffic online and argued that faster action is needed to enable a sustainable ecosystem.
Cloudflare explicitly referenced the “world’s largest search engine” — an apparent nod to Google — and claimed that it has access to roughly “2x more information” than other AI companies because of how it makes discoverability and AI use interact. Google has previously contested such generalizations, citing a bot called Google Extended that allows site owners to opt out of having content used for training and AI products like Gemini Apps and Vertex API without affecting inclusion in Google Search. Google’s main crawler, Googlebot, continues to index for Search and supports features such as AI Overviews and AI Mode.
Payment options and publisher tools
In recent years Cloudflare has rolled out tools aimed at giving publishers more control over how AI systems access their content, including a marketplace that enabled sites to charge AI bots for scraping, previously called Pay Per Crawl. The company said this model is evolving into “Pay Per Use,” which would let publishers charge AI companies when the publisher’s content creates value for the AI service, not only when the content is fetched.
Cloudflare’s data indicated that more than 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages, a rationale cited for conserving publishers’ bandwidth and compute resources.
To implement paid access, Cloudflare is initially working with two partners: Ceramic.ai and You.com. When a publisher opts in, they receive payment when their content appears in Ceramic’s AI search results or when You.com accesses a piece of their premium content. Cloudflare says other AI companies can adapt this model to their own needs.
Summary
The September 15, 2026 default change aims to more clearly separate traditional search crawling from AI training and agent use, while giving website owners greater control and commercial options. The change affects new customers, newly created sites of existing customers and all existing free customers, and is paired with a shift toward a Pay Per Use model and initial partnerships with Ceramic.ai and You.com.
