A toggle on a browser window branches to four web crawler icons.

OpenAI Cross-Company Web Crawler Signal: What It Is, How It Works, and Who It Affects

ByAIERA Blog Editorial Team

Key Takeaways

  • On 30 October 2026, OpenAI said it will honour a common robots.txt signal for AI training preferences, alongside Google, Microsoft and Anthropic — one shared rule instead of a separate rule for each company's crawler.
  • The signal is the policy token __usage_ai-training, from an Internet-Draft by Kevin Kelley of Anthropic. It is a proposal, not a finished standard, and it expires on 3 May 2027 unless it is revised or adopted.
  • OpenAI has given no firm date: it aims to implement the signal "in the coming months". Anthropic has already started, Microsoft starts 1 February 2027, and Google aims for 2027.
  • For OpenAI specifically, the signal sits alongside three separate crawlers with separate jobs: GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for on-demand fetches. The new token covers training only — it does not affect ChatGPT search answers.
  • OpenAI has not said what happens when the shared token conflicts with an existing GPTBot rule. Until it does, publishers should keep their GPTBot rules in place.
  • Meta is the notable holdout: it has praised the idea but has not committed to honouring the token, so the signal does not yet cover every major lab.

Why This Story Is About OpenAI

On 30 October 2026, four companies announced the same thing on the same day: OpenAI, Google, Microsoft and Anthropic each said they will honour a single shared signal in robots.txt that lets a website state its preference about AI training. Our earlier article covered that four-company announcement as a whole. This article is about one company's side of it — OpenAI's — because OpenAI is the name most publishers, site owners and search professionals are actually asking about, and because OpenAI's position has the most unanswered questions.

The reason is simple. OpenAI runs the crawlers that matter most to the largest number of websites. GPTBot is the crawler that collects content which may be used to train OpenAI's models. OAI-SearchBot is the crawler that decides whether a site appears in ChatGPT search answers. ChatGPT-User is the fetcher that opens a page when a person asks ChatGPT to read it. Millions of sites have rules for one or more of these already, and the question every one of them is asking is the same: does the new shared signal replace my OpenAI rules, sit alongside them, or conflict with them? OpenAI's announcement, as published, does not fully answer that question. This article sets out what OpenAI has said, what its crawlers actually do, what the shared signal changes and what it leaves unchanged, and what a site owner should do now.

The announcement itself was a short post from OpenAI, published on 30 October 2026 and titled "Making it easier for websites to express AI training preferences". Its core sentence is that OpenAI "will honor a common signal in robots.txt" for preferences about foundational generative AI training. OpenAI called the arrangement a voluntary effort that "makes it easier for publishers and website operators to communicate the same preference to multiple participating providers at once". It said it aims to implement the signal "in the coming months" and will share details when it does. And it said the new signal will not replace its existing robots.txt crawler controls, which will continue to apply.

That last sentence is the one that matters most for anyone running a website, and it is also the one that creates the most confusion. If the old controls stay, and the new signal is added, what happens when they disagree? OpenAI has not said. That gap is the central unresolved question of this article, and it is worth keeping in mind through everything that follows.

The Signal Itself: One Token, Four Companies

The shared signal is a policy token called __usage_ai-training. It comes from an Internet-Draft titled "Machine-Readable Training Preference Signal", authored by Kevin Kelley, a technical lead for standards at Anthropic, and dated 30 October 2026 — the same day as the four announcements. The draft was submitted to the Internet Engineering Task Force's AI Preferences working group, known as AIPREF, which has spent more than a year working on a common language for AI usage preferences.

The mechanism is deliberately small. It adds no new commands and no new grammar to robots.txt. Instead, it defines a token that goes in the User-Agent line of an ordinary robots.txt group. A site that wants to opt out of training across the whole site writes: User-Agent: __usage_ai-training — Disallow: /

A site that wants to opt out of most of the site but keep one section available — a press kit, for example — adds an Allow line for that folder: User-Agent: __usage_ai-training — Disallow: / — Allow: /press-kit/

The token must match exactly, apart from letter case. A catch-all User-Agent: * group does not count as naming the token, so a site that only has a wildcard rule has expressed no training preference at all under this mechanism. That is a detail that will catch people out: the shared signal only works if the token is written out in full.

The draft's definition of training is borrowed from the AIPREF working group's vocabulary: using an asset "to modify the learned parameters of a generative AI model". That definition sets the scope of the whole mechanism. It covers training only. It does not cover search indexing, and it does not cover grounding — the step where an AI tool reads a live page to answer a question in real time. Those are different uses, governed by different controls, and confusing them is the easiest way to set the wrong preference.

The draft's status matters as much as its content. It is an individual Internet-Draft, which means it is a proposal put forward by one author, not a standard endorsed by the IETF. The IETF datatracker states plainly that an individual draft "is not endorsed by the IETF and has no formal standing in the IETF standards process". The draft expires on 3 May 2027 unless it is revised or adopted, which gives the working group a deadline to keep refining it. The token name and details may still change before anything becomes a standard. OpenAI's commitment is therefore a commitment to honour a proposal that may still change shape — a reasonable thing to do, but not the same as adopting a finished standard.

OpenAI's Three Crawlers, and Why the Distinction Matters

To understand what the shared signal does and does not do for OpenAI, you need to understand OpenAI's crawlers, because OpenAI does not run one crawler. It runs several, each with a separate job, and each with its own rule in robots.txt. OpenAI's own crawler documentation states that each setting is independent of the others. That independence is the key to the whole story.

GPTBot is the training crawler. OpenAI describes it as used "to crawl content that may be used in training our generative AI foundation models". Its current user agent is GPTBot/1.4. Disallowing GPTBot tells OpenAI not to use a site's content for that training. This is the crawler the shared signal is about: when OpenAI implements the __usage_ai-training token, it will be read as a training preference of the same kind that a GPTBot disallow expresses today.

OAI-SearchBot is the search crawler. It builds the index behind ChatGPT search and decides whether a site's pages can be shown and cited in ChatGPT search answers. Its current user agent is OAI-SearchBot/1.4. OpenAI states that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links". Blocking it does not remove a site from training, and allowing it does not put a site into training. The two settings are independent, and OpenAI's own example is to disallow GPTBot while allowing OAI-SearchBot — keeping content out of model training while staying eligible for ChatGPT search.

ChatGPT-User is different again. It is not a crawler in the usual sense. It fires only when a person asks ChatGPT or a Custom GPT to read a specific page, or interacts with a Custom GPT through GPT Actions. OpenAI is direct about what that means for robots.txt: "Because these actions are initiated by a user, robots.txt rules may not apply." A site cannot reliably use robots.txt to stop ChatGPT-User from opening a page a user has asked about, because a person started the request. The shared training token does not change that either — it is a training preference, not a fetch block.

OpenAI also lists a fourth bot, OAI-AdsBot, which checks pages submitted as ads on ChatGPT. OpenAI says its data is not used for training. It is irrelevant to the shared signal, but it is worth knowing it exists so that it is not confused with the other three.

The practical consequence of this split is that the shared signal, when OpenAI implements it, will govern only the training side of OpenAI's relationship with a site. It will not change whether a site appears in ChatGPT search answers, it will not stop ChatGPT-User from fetching a page a user asks about, and it will not remove content from models that have already been trained. A publisher who wants to control all three needs three different settings, and the shared token is only one of them.

What OpenAI Has Said, and What It Has Not Said

OpenAI's announcement is short, and its gaps are as informative as its commitments. Here is what is established, and what is not.

What is established: OpenAI will honour a common signal in robots.txt for foundational generative AI training. It aims to implement the signal "in the coming months" and will publish implementation details when it does. The signal will not replace OpenAI's existing crawler controls, which will continue to apply and be respected. The commitment is voluntary, not a legal obligation, and it covers foundational generative AI training — the training of the base models — rather than every possible use of web content.

What is not established: OpenAI has given no date, not even a month. "In the coming months" could mean the end of 2026 or well into 2027, and OpenAI has said nothing more specific since the announcement. OpenAI has not said what happens when a site's shared rule conflicts with its existing GPTBot rule — for example, a site that disallows GPTBot but allows the shared token, or the reverse. Coverage of the announcement in Search Engine Journal, cited in reporting on the launch, notes this as an open question. OpenAI has not said how a publisher will verify that its preference has been applied, or how to audit whether content was used for training after the fact. And OpenAI has not said whether the shared token will be read by all of its services at once, or rolled out service by service.

The conflict question deserves emphasis, because it is the one most likely to cause real problems. Imagine a site that blocked GPTBot two years ago and has never touched its robots.txt since. When OpenAI starts reading the shared token, that site has no shared-token rule at all, so the draft's position is that it expresses no preference under the new mechanism — but its GPTBot rule still stands, and OpenAI says existing controls will continue to apply. That site is probably fine. Now imagine a site that adds the shared token to opt out of training, but has an old GPTBot rule that allows crawling, left over from a time when the site wanted to be in training data. Which rule wins? OpenAI has not said. Until it does, the safe course is to make the two rules agree: if you want out of training, disallow GPTBot and add the shared token. Do not rely on one rule to override the other.

The other three companies have been more specific, which throws OpenAI's vagueness into relief. Anthropic, whose standards lead wrote the draft, says it has already started implementing support, though rollout across its services will take time. Microsoft says it will start honouring the new preferences on 1 February 2027 as Bing recrawls pages, and it is adding a separate page-level control for AI grounding, plus a meta tag or HTTP header for single pages. Google says it aims to support the token in 2027, with its crawler documentation pointing to early 2027, and it has said it will treat a shared opt-out as if the site had disallowed Google-Extended, its own AI training control since 2023. OpenAI's "in the coming months" is the least specific of the four, and it is the one publishers most need to pin down.

Who It Affects, and How Much It Changes for Them

The shared signal affects different groups in different ways, and the size of the change depends on who you are.

For publishers and news sites, it is a genuine simplification — eventually. A publisher that currently maintains separate rules for GPTBot, Google-Extended, ClaudeBot and Microsoft's tags will, once all four companies implement the token, be able to express the same preference once. That is real progress, and it is why the coordination matters even though the mechanism is still a proposal. But the simplification is not available yet at OpenAI, and it will not be available until OpenAI implements the signal and confirms the details. Until then, the old rules are the ones that work, and removing them would be a mistake.

For small site owners and bloggers, the change is smaller but still useful. Most small sites have never written a GPTBot rule at all, and many do not know robots.txt exists. For them, the shared token is a single, copyable block that covers four companies at once — once it is live. The risk is the opposite one: a small site owner who reads a headline, adds the token, and assumes the job is done, when in fact OpenAI does not read the token yet and the site's content remains available to GPTBot. The token is a forward declaration, not an instant switch, and it should be presented that way to anyone who is not technical.

For SEO professionals and agencies, the signal creates a new advisory task. Clients will ask whether to opt out, and the honest answer is that it depends on what the client wants from AI. A site that opts out of training through GPTBot and the shared token does not lose its place in ChatGPT search — that is governed by OAI-SearchBot, a separate setting — but a site that confuses the two and blocks OAI-SearchBot by mistake will lose its citations in ChatGPT search answers. The distinction between training and search is the single most important thing to get right in client advice, and it is the distinction most likely to be lost in a headline.

For ChatGPT users, the signal changes almost nothing directly. It does not change what ChatGPT can answer, it does not change ChatGPT search, and it does not change what happens when a user asks ChatGPT to read a page. Its effect is upstream, on what goes into future models, and it will not be visible in the product for months or years.

For the AI labs themselves, the signal is a coordination achievement with a conspicuous gap. Meta, the fifth major lab, is not among the four. Meta's communications head told The Verge that creating a common signal is "an important step", but the company did not say whether it would honour the token. That matters because Meta's crawlers are a separate route into training data, and a single rule that covers four labs but not the fifth is progress, not a universal rule. Meta is also moving in the other direction on commercial terms: its new business terms would let it crawl sites to train its AI with no opt-out, and on 30 October 2026 — the same day as the shared-signal announcement — the UK Competition and Markets Authority opened an investigation into whether those terms could force publishers to let Meta use their content without a meaningful choice. Meta paused the rollout pending consultation. No finding of illegality has been made. The two developments on the same day are not evidence of one coordinated policy, but they show the two routes to the same question: a voluntary signal on one side, and a regulator's investigation on the other.

What to Do Now: The OpenAI Checklist

For a site owner who wants to act on this, the steps are small, reversible, and worth doing in order.

First, check what you already have. Open your robots.txt file and look for rules naming GPTBot, OAI-SearchBot and ChatGPT-User. If you have a GPTBot disallow that is working, leave it alone. It is the control that actually functions today, and OpenAI says it will continue to apply.

Second, add the shared token now, in the draft's format, with any Allow exceptions you genuinely want. Adding it early costs nothing and means the preference is already in place when OpenAI switches on support. But do not remove the GPTBot rule when you add it. The shared token is a promise about the future; the GPTBot rule is the control that works now.

Third, make the two rules agree. If you want out of training, disallow GPTBot and disallow the shared token. If you want to stay in training data, allow both. Do not set them in opposite directions and hope OpenAI resolves the conflict in your favour, because OpenAI has not said how it will resolve it.

Fourth, leave OAI-SearchBot alone unless you have a specific reason to block it. Allowing OAI-SearchBot is what keeps a site eligible for ChatGPT search answers, and blocking it by mistake is the most common way to lose visibility in AI search. OpenAI says robots.txt changes take about 24 hours to reach its search systems, so a mistake here is reversible within a day — but it is still a mistake worth avoiding.

Fifth, remember what the token does not do. It does not remove content already collected into datasets or trained into models. It does not stop ChatGPT-User from fetching a page a user asks about. It does not cover Meta's crawlers. And it does not create a receipt, a registry or an audit trail. There is no way, at present, for a publisher to verify that every service has applied its preference. That verification gap is the mechanism's biggest weakness, and no announcement has closed it.

What We Could Not Verify

Several things about OpenAI's side of the signal remain unverified, and they should be stated plainly.

OpenAI has not published a date for implementation, and no date can be inferred from its post. Any date attached to OpenAI's support for the token, other than "in the coming months", is a guess. OpenAI has not said how it will resolve a conflict between the shared token and an existing GPTBot rule, and no resolution can be assumed. OpenAI has not published details of how the token will be read — whether by GPTBot itself, by a separate process, or by each service independently — and no implementation detail should be stated as fact until OpenAI publishes it.

The IETF draft itself may change. It is an individual draft that expires on 3 May 2027, and the token name, the scope, or the wording on verification could all change before it becomes a standard. A site that adds the token today should check the draft's status again in 2027 rather than assuming the current wording is final.

Finally, the claim that all four companies will honour the token rests on their announcements, not on observed behaviour. Anthropic says it has started implementing; the other three have not yet switched anything on. Whether the token works as promised will only be known when each company confirms its support in its own documentation and when publishers can observe the effect. Until then, treat the shared signal as a useful, low-cost step — and keep the controls that already work.

For the four-company picture, including the timelines for Google, Microsoft and Anthropic and the details of Microsoft's separate grounding control, see our earlier article on AI training opt-out. For the parallel policy track, see our coverage of the EU AI copyright consultation, and for how AI search and grounding differ from training, see our guide to what AI agents are and how they work.

Sources and Further Reading

  • OpenAI, "Making it easier for websites to express AI training preferences", 30 October 2026, as reported in coverage below.
  • IETF Internet-Draft, "Machine-Readable Training Preference Signal" (draft-kelley-aipref-training-preference-signal-00), 30 October 2026.
  • TSN Media, "AI Training Opt-Out: Google, OpenAI, Microsoft Sign On": https://tsnmedia.org/ai-training-opt-out-robots-txt/
  • Barista Labs, "Robots.txt AI training opt-out gets one shared token": https://www.baristalabs.io/blog/robots-txt-ai-training-shared-token
  • Derrington Collaborative AI, "One shared robots.txt token to opt out of AI training": https://www.derringtoncollaborativeai.com/briefing/shared-ai-training-opt-out-token
  • OpenAI, "Overview of OpenAI Crawlers", as summarised in: https://icoda.io/blog/gptbot-guide/

Similar Posts