OpenAI, Google, Microsoft and Anthropic announced a shared technical approach intended to let publishers prevent artificial intelligence systems from collecting data from their sites for model training. The proposal aims to avoid the need for publishers to set separate restrictions for each individual AI bot.
What the proposal does
The solution centers on adding a so-called "policy token" to a website's robots.txt file. The robots.txt file is an established mechanism for indicating which bots may crawl or index a site and which may not — for search indexing, archiving, or other automated access. With the new token, a single opt-out signal could be recognized by multiple AI services rather than requiring per-bot configuration.
Why this matters
Training modern AI models often involves automated crawling of public web content, which can increase traffic and impose additional server load and costs on content hosts. The announcement notes that some projects, such as Wikipedia, have experienced significantly higher server traffic related to automated crawling for AI training, creating financial and operational strain for maintainers.
Under current practice, websites typically must indicate separately for each AI bot if they do not want their content used for model training. The proposed standard would let publishers express a general refusal once, and have that refusal respected across participating AI providers.
Potential impacts
If adopted widely, the standard could reduce unnecessary bandwidth and hosting costs for content providers and establish a clearer technical norm for how AI services should respect publisher preferences. The ultimate effect will depend on how broadly the token is supported in practice and on cooperation between AI companies and content hosts.
Summary
OpenAI, Google, Microsoft and Anthropic propose using a robots.txt "policy token" to enable a single, consistent opt-out from AI training data collection for publishers. The approach targets growing operational burdens on websites caused by automated AI crawling and seeks to simplify how publisher preferences are signaled and enforced.
