AI Training Opt-Out: One Rule for Four AI Labs
Key Takeaways
- AI training opt-out is now, in principle, a one-line job: on 30 October 2026, OpenAI, Google, Microsoft and Anthropic said they will honour a single robots.txt policy token, `__usage_ai-training`, for training.
- The token covers training only. It does not block crawling, does not remove a site from search results, and does not pull content out of datasets or models that already exist.
- Timelines are uneven: Anthropic says it has begun implementation, OpenAI says “in the coming months”, Microsoft targets 1 February 2027, and Google aims for 2027.
- The mechanism is an individual Internet-Draft, not a finished standard. It was submitted to the Internet Engineering Task Force (IETF) on 30 October 2026 and, as written, expires on 3 May 2027 unless it is revised or adopted.
- For now, keep the company-specific controls that already work — such as Google-Extended — alongside the new token, and treat the shared signal as a forward-looking preference, not an instant switch.
Introduction
For years, telling AI companies not to train on your website has meant learning a different dialect for each one. Google had Google-Extended. OpenAI had GPTBot. Anthropic had ClaudeBot. Microsoft had its own tags. A publisher who wanted a consistent answer had to maintain a small stack of rules, check each company’s documentation, and hope nothing changed quietly in between.
On 30 October 2026, the four largest Western AI labs announced, in coordinated posts, that they will honour one shared signal instead. A site owner writes a single policy token in robots.txt — `User-Agent: __usage_ai-training` followed by `Disallow: /` — and, as support arrives, all four companies are supposed to read it the same way: do not use this content for AI training. It is the first time the major labs have aligned on a single opt-out mechanism, and that coordination is the real news, even though the mechanism itself is still a proposal and mostly not yet live.
This article explains what was announced, how the token works, what each company’s timeline actually is, what the signal does not do, and what a site owner, creator or small business should do this week. The important discipline is to separate the promise from the plumbing. The promise is simple and welcome. The plumbing — dates, scope, verification and legal standing — is where the caveats live.
What was announced on 30 October 2026
OpenAI, Google, Microsoft and Anthropic each said they will honour the shared token. Coverage of the announcement, including reporting in The Verge cited by AI Weekly and Skimwire, describes it as a single robots.txt policy token that lets a publisher opt out of training across every participating company’s crawlers in one directive.
The token comes from an Internet-Draft titled “Machine-Readable Training Preference Signal”, authored by Kevin Kelley of Anthropic and dated 30 October 2026. The draft was submitted to the IETF’s AI Preferences (AIPREF) working group. That detail matters because it sets the status of the whole mechanism: an individual Internet-Draft is a proposal put forward by one author, not a standard endorsed by the IETF. The IETF datatracker notes that an individual draft “is not endorsed by the IETF and has no formal standing in the IETF standards process.” The draft expires on 3 May 2027 unless it is updated or adopted, which means the working group has a deadline to keep refining it.
Google’s Search Central post, by Search Advocate John Mueller, frames the token as a shared AI training control and says Google aims to support it “along with other AI providers in 2027”. Microsoft’s Bing Webmaster post gives the firmest date. OpenAI’s post says it will honour a shared robots.txt signal to help website operators communicate their training preferences to participating providers, with reporting putting its implementation in “the coming months”. Anthropic, as the draft’s author, has said it has started implementing support, while noting that rollout across its services will take time and that its interpretation may change as the standard evolves.
Meta, notably, is not among the four. Reporting on the announcement says Meta supports the idea but has not committed to implementing it. That is a significant gap for publishers, because Meta’s crawlers are a separate route into training data. A single rule that covers four labs is progress, but it is not yet a universal rule.
How the token works, in plain terms
Robots.txt is a plain-text file at the root of a website that tells automated crawlers which parts of a site they may visit. It is a voluntary convention, not access control: well-behaved crawlers respect it, but it does not physically block anyone. The new token works inside that same convention, with one twist. Instead of naming a crawler, the site names a usage.
The basic block, as given in the draft and in coverage, is:
“`
User-Agent: __usage_ai-training
Disallow: /
“`
A site can also carve out exceptions with `Allow:` lines — for example, disallowing training for the whole site except a press kit folder. Google has confirmed that if a site uses the shared token without explicitly naming Google-Extended, Google will interpret it as if the publisher had disallowed Google-Extended. That mapping is useful, because it means existing Google-Extended users do not have to run two contradictory rules once Google’s support lands.
The most common misunderstanding to avoid is that this token is a crawl block. It is not. The draft itself states that the policy token “is not a directive for crawling behavior”. A crawler may still visit a page, fetch it, and cache it; what the token governs is whether the content may be used for generative-AI training. That distinction is deliberate. It is also why the token, on its own, does not change search indexing or AI-generated answers that quote a page.
Microsoft goes further than the shared token in one respect. It plans a separate control for AI grounding — the retrieval step that pulls live web content into chat answers, rather than training model weights. Microsoft’s schedule for that control runs from 15 December 2026, when temporary transition settings open in Bing Webmaster Tools, through 30 September 2027, when those temporary settings expire. Microsoft also says its new preferences replace its older NoArchive and NoCache tags, and that sites using those tags should set the new preferences by 1 March 2027, after which the old tags are no longer read. Publishers who rely on Microsoft’s tags should read that transition carefully rather than assuming a one-line change covers everything.
When each company actually starts
This is where the announcement needs the most careful reading, because “we will honour it” does not mean “we honour it today”.
The practical consequence is a staggered start across the first half of 2027, at best. A publisher who adds the token on 31 October 2026 should expect it to do nothing at most of the four companies for weeks or months, and should not remove existing per-company rules until each company’s support is confirmed. The new token is best understood as a forward declaration: put it in place now so that, as each lab switches on support, the preference is already there to be read.
There is also a verification gap that no announcement has closed. The announcement does not spell out how a publisher will verify that every service has applied its preference, or how to audit a crawler’s training use after the fact. Robots.txt has always relied on good faith and on crawler behaviour that can be observed only indirectly. The shared token improves the signalling problem — one rule instead of four — but it does not solve the enforcement problem. Publishers should not present it to clients or readers as a guarantee.
What the token does not do
Four limits deserve to be stated plainly, because each one corrects a plausible over-reading of the headlines.
First, it does not remove content already collected. Updating robots.txt does not remove material already in existing datasets or trained into model weights. Models already trained on a site’s pages are not affected, because the draft judges preferences at fetch time. A publisher who wants older material addressed still needs a separate route — a takedown request, a licensing negotiation, or litigation — and none of those is created by this token.
Second, it does not cover AI answers. Opting out of training will not keep a page out of AI-written search answers, AI overviews, or grounded chat responses that quote live pages. Those uses are governed by different controls, including Microsoft’s separate grounding preference and the existing search and AI-answer settings each company offers. A site that wants to control both training and grounding needs to set both, where the controls exist.
Third, it does not affect search ranking or inclusion. Google says Google-Extended, which the new token maps to, does not affect inclusion in Google Search and is not a ranking signal. Microsoft says neither its training nor its grounding preference changes eligibility for traditional Bing results. Publishers who feared that opting out of training would cost them search traffic can set that fear aside, at least on the companies’ stated positions.
Fourth, it is not law, and it is not a finished standard. It is a voluntary, company-by-company commitment to honour a proposal that may still change. Its legal weight, if any, would come from a company’s own published commitments and from applicable law in each jurisdiction, not from the draft itself. That is also why the parallel legal track matters: on 30 October 2026, the Wikimedia Foundation submitted an amicus brief in the New York Times case against OpenAI, and Meta postponed the effective date of its updated commercial terms — which publishers had argued would broaden Meta’s rights over their content — from 30 October to 20 November 2026, after complaints to antitrust authorities in the European Union and the United Kingdom. The shared token and those disputes are different routes to the same question: who gets to decide how published work is used to train AI.
What to do this week
For most site owners, the sensible steps are small and reversible. Add the shared token to robots.txt now, in the draft’s format, with any `Allow:` exceptions you genuinely want. Keep every existing per-company rule that currently works — Google-Extended, GPTBot, ClaudeBot and Microsoft’s tags — until each company confirms its support for the shared token and you have checked its documentation. Do not delete a working control in favour of a promised one.
If you run a site with sections you do want in training data — for example, a press kit, documentation you want models to know about, or public reference material — use the `Allow:` mechanism deliberately rather than opting the whole site in or out by accident. And if you advise clients, be explicit about the timeline: the token is a preference that takes effect as each lab implements it, starting in early 2027 for Microsoft and Google, and it does not retroactively clean training data.
For creators and small publishers, the deeper question is whether a voluntary signal is enough. The labs’ coordination is genuinely new, and a single standard, if it survives the IETF process, would lower the cost of expressing a preference for everyone. But the history of robots.txt is a history of good-faith compliance by major crawlers and non-compliance by everyone else. The shared token does not bind crawlers outside the four companies, and it does not create a registry, a receipt, or an audit trail. Whether it becomes more than a polite request will depend on whether the IETF process produces a stable standard, whether Meta and other labs join, and whether regulators or courts give the signal legal weight. Until then, treat it as a useful, low-cost step — and keep the other tools in the box.
Our earlier coverage of the EU AI copyright consultation closing on 3 November sets out the parallel policy track, where creators and AI firms are being asked to respond before a deadline rather than relying on a voluntary signal alone. And our guide to what AI agents are and how they work explains why the distinction between training a model and grounding an answer in live pages matters so much for publishers: the two uses look similar from the outside, but they are governed by different controls, and confusing them is the easiest way to set the wrong preference.
What to watch next
Watch three things. First, the IETF process: whether the draft is revised before its 3 May 2027 expiry, whether it gains working-group adoption, and whether its wording on scope and verification changes. Second, implementation dates: OpenAI’s “coming months”, Google’s 2027 window, and Microsoft’s 1 February 2027 start, each confirmed in the company’s own documentation rather than in a press summary. Third, who joins: Meta’s decision, and whether other model providers — including open-weight labs and smaller crawlers — adopt the token or offer their own incompatible version. If the token fragments again, its main advantage disappears.
For now, the fair summary is this: four major labs have agreed, for the first time, to read the same opt-out sentence the same way. That is real progress for publishers who have spent years maintaining four different rules. It is not yet a standard, not yet live at most of the companies, and not a remedy for training that has already happened. Add the line, keep the old rules, and judge the mechanism by what each company actually does in 2027.
