OpenAI, Google, Microsoft and Anthropic announced a joint standard designed to let publishers declare that they do not want artificial intelligence systems to collect data from their sites. The proposal adds a so-called "policy token" to websites' robots.txt files to indicate this preference.
What the policy token does and how it works
The robots.txt file has long been used to tell automated agents which parts of a site may be indexed or archived. Under the new standard, a policy token placed in robots.txt would provide a single, consistent signal that multiple AI bots can respect, removing the need for publishers to configure blocking individually for each provider.
According to the announcements, Microsoft says its bots will honor the new signal, while noting that the opt-out will not affect whether a page appears in Bing search results. The token is specifically intended to prevent AI systems from using site content for model training.
Why the standard is timely
The emergence of modern AI models has led to a significant increase in bots crawling the web for training data, imposing bandwidth and cost burdens on publishers and platform operators. The article highlights Wikimedia's experience, where server traffic from such bots rose substantially, increasing operating costs and straining the project's finances.
Today, publishers use different approaches to address bot-related issues. One option is Really Simple Licensing (RSL), which lets publishers charge fees for bot access; another approach, cited in the article, is that providers like Claudflare completely block bots from collecting data.
Rollout timeline
Anthropic has already implemented the new signal. Google and Microsoft plan to begin supporting it next year, while OpenAI says it will start enforcing monitoring of the signal "in the coming months."
Implications for publishers and AI developers
A unified, robots.txt-based signal could reduce administrative overhead for publishers by allowing them to express a single preference about AI training access. However, the announcements do not spell out every technical or legal consequence—for example, how cases will be handled where a publisher opts out of AI training but a search engine still displays the page in results.
Ultimately, the effectiveness of the standard will depend on broad adoption and practical implementation; the next several months and the coming year will determine how this change affects the availability of web content for AI training.
