OpenAI announced it will honor a common signal placed in robots.txt files that allows website operators and publishers to communicate, to multiple AI providers at once, their preferences about whether their content may be used to train foundational generative AI models. The company described the initiative as voluntary and intended to reduce administrative overhead for site operators.
Until now, website operators typically needed to identify individual crawlers and maintain separate rules for each provider. A shared signal enables multiple participating providers to observe the same setting, simplifying rule management for site owners.
OpenAI said it plans to implement the signal in the coming months and will provide an additional update with implementation details when available. The company emphasized that the new signal will not replace existing robots.txt controls: current crawler controls will continue to apply and be respected.
The move builds on work by the Internet Engineering Task Force (IETF) AI Preferences working group. OpenAI is participating in that working group and joined other participants in announcing the shared signal. The company stated it will continue to adapt its practices as IETF discussions and the broader legal and technical landscape evolve, with attention to appropriate controls and fairness.
OpenAI framed the change as a practical step toward preserving an open web where people can publish and discover information, while giving website owners clearer and simpler means to express how their content may be used. The company encouraged other AI providers to honor the same signal.
