OpenAI, Google, Microsoft, and Anthropic Back Shared robots.txt Token for AI Training Opt-Out
OpenAI, Google, Microsoft, and Anthropic announce support for a shared robots.txt token allowing publishers to opt out of AI training across all four labs.
In a historic collaborative agreement announced October 30-31, 2026, OpenAI, Google, Microsoft, and Anthropic confirmed they will jointly honor a unified robots.txt policy token: __usage_ai-training. The shared directive enables web publishers to opt out of AI foundation model training across all four major labs simultaneously without affecting web search indexing.
Unified robots.txt Policy Token Across Four Frontier Labs
On October 30-31, 2026, OpenAI, Google, Microsoft, and Anthropic announced an unprecedented coordinated policy to honor a single, standardized directive in website robots.txt files: User-Agent: __usage_ai-training followed by Disallow: /. Authored as an IETF Internet-Draft by Anthropic standards lead Kevin Kelley and submitted to the IETF AIPREF working group, the standard allows content creators and digital publishers to block automated training data ingestion across all four major Western AI labs using a single consolidated rule.
Decoupling Training Permission from Web Search Indexing
A critical technical distinction of the new token is that it governs training permissions rather than basic crawling behavior. Web crawlers like Googlebot, Bingbot, and GPTBot may still visit, index, and cache web pages for search results and retrieval-augmented generation (RAG) citations, ensuring sites remain discoverable in search engines while explicitly withholding consent for model weight pretraining. Google confirmed that sites implementing the shared token will automatically be treated as opting out of Google-Extended.
Implementation Timelines and Industry Standardization Limits
Implementation schedules vary among the four participants: Anthropic activated immediate support upon announcement, OpenAI committed to deployment in the coming months, Microsoft targeted February 1, 2027 (alongside separate controls for AI grounding), and Google slated full rollout for 2027. While backed by stakeholders including Meta, Cloudflare, Common Crawl, News Corp, the BBC, and the Electronic Frontier Foundation, the token functions as a voluntary coordination mechanism under RFC 9309 without automated enforcement, and applies strictly to prospective training runs rather than retrospectively purging existing training datasets.