Big AI Labs Back Shared robots.txt Token for Publisher

OpenAI, Google, Microsoft and Anthropic announced on October 30, 2026 that they will honor a single robots.txt policy token, __usage_ai-training, that allows website publishers to opt out of AI model training across all four crawlers in one directive. The move represents the first coordinated attempt by major Western AI labs to standardize publisher preferences for training data collection, though timelines for implementation vary and the mechanism carries no enforcement power.

The shared token is the product of an individual IETF Internet-Draft authored by Anthropic standards lead Kevin Kelley, dated October 30, 2026, with an expiry of May 3, 2027. While the document has been submitted to the IETF's AIPREF working group, chaired by Cloudflare's Mark Nottingham and Cisco's Suresh Krishnan, it remains an individual draft with no formal standing in the IETF standards process and no endorsement from the body itself.

How the Shared Token Works

A site owner writes User-Agent: __usage_ai-training followed by Disallow: / in their robots.txt file to block all four labs from using site content for model training. Publishers can carve out exceptions with Allow: lines for specific paths. Google's John Mueller confirmed that if a site uses the shared token without explicitly naming Google-Extended, Google will interpret the directive as if the publisher had disallowed Google-Extended.

The token controls training permission, not crawling behavior. As the draft states, "This policy token is not a directive for crawling behavior," meaning bots may still fetch and index the page for search or other purposes. The distinction is critical: opting out of training does not prevent a crawler from visiting or caching a page.

Implementation Timelines Vary Across Labs

Anthropic has already begun implementing support for the token as of the October 30 announcement. OpenAI stated it aims to implement the signal in the coming months, without committing to a specific date. Microsoft has set a target of February 1, 2027, and Google has committed only to 2027 without further detail.

Microsoft is adding a granular layer beyond the shared token: a separate control for AI grounding, the retrieval step that pulls live web content into chat answers rather than training model weights. This distinction allows publishers to permit training while blocking real-time citation, or vice versa, though the technical specification for that additional control has not been published.

Officially Launched on October 30, 2026

The announcement was made public on October 30, 2026, with The Verge reporting the coordinated statements from all four labs. Participants in the IETF working group include Meta, Cloudflare, the Common Crawl Foundation, News Corp, the Financial Times, the BBC, the Electronic Frontier Foundation and Creative Commons. The breadth of stakeholders signals industry interest, though the draft's individual status means no participant is bound to comply.

Two Hard Limits on Effectiveness

OpenAI framed the mechanism as "a coordination mechanism rather than a security control" under RFC 9309, which means non-compliant scrapers and smaller labs can disregard the directive with no technical consequence. The token relies entirely on voluntary compliance.

More importantly, updating a robots.txt file does not remove content that has already been collected into existing datasets or trained into model weights. Publishers who opt out today are only blocking future training runs, not reversing past ingestion. This limitation has been a recurring friction point in publisher negotiations with AI labs, and the new token does nothing to address it.

What This Means

The shared robots.txt token represents a rare moment of cross-lab alignment in a policy landscape that has been fragmented and adversarial. For publishers, it reduces the overhead of managing multiple per-lab opt-out mechanisms. For the labs, it provides a single interface to point to when facing criticism over training data practices. But the mechanism's lack of enforcement power, its inability to remove already-trained material, and the staggered implementation timeline mean publishers will still need to monitor compliance crawler by crawler. The token is a coordination step, not a solution.