Engineering

Can you stop AI companies training on your website?

Four AI companies now back one robots.txt line for opting out of training. Here is what it does, what it does not do yet, and copy-paste rules for a small business site.

By Shashi Kumar ·

Illustration of a website globe with a robots.txt file standing guard between it and a row of AI crawler bots

For years the honest answer to "can I stop AI companies training on my website?" was: sort of, one company at a time. You had to learn a different bot name for every lab, and hope each one kept its promise.

That changed a little on 30 October 2026. Google, Microsoft, OpenAI and Anthropic each said they will honour one shared line in robots.txt that means "do not train AI on this". It is a real step forward. It is also not switched on everywhere yet, and it does nothing about whether your site shows up in AI search answers, which for many small businesses is the part that actually matters.

What was announced, exactly

The new signal is a policy token called __usage_ai-training. It is defined in an IETF Internet-Draft, Machine-Readable Training Preference Signal, written by Kevin Kelley of Anthropic and published on 30 October 2026. You use it like any other robots.txt group:

User-Agent: __usage_ai-training
Disallow: /

A few details from the draft are worth knowing:

  • It is an individual draft. The IETF itself says it has no formal standing yet, and it expires on 3 May 2027 unless revised.
  • A User-agent: * group never counts as this token. If you want to opt out of training, you have to name it.
  • It is about use, not access. The draft says the token "is not a directive for crawling behavior". Bots still visit; the token says what they may do with what they read.

Who has committed, and when

As of 9 November 2026, nobody has flipped this on across the board. The timelines differ:

CompanyWhat they saidWhen
AnthropicHas started implementing it; rollout is gradualFrom 30 Oct 2026
OpenAIWill honour it; current robots.txt rules stay in force meanwhile"Next few months"
Microsoft (Bing)Training opt-out via the token, plus new page-level grounding controlsFrom 1 Feb 2027
GoogleWill treat the token as a Google-Extended opt-outEarly 2027

Sources: Google, Microsoft, Anthropic (below) and Search Engine Journal for OpenAI's timing.

Two details shape how you write your file. Anthropic says a rule that names ClaudeBot takes priority over the shared token. Google says that if you use the token and do not mention Google-Extended, it will act as if you had disallowed Google-Extended. So the token is the default, a vendor-specific rule overrides it, and you should keep your existing bot rules for now.

robots.txt in two minutes

robots.txt is a plain text file at yoursite.com/robots.txt that bots read before crawling. Three things trip people up:

  1. It is a request, not a lock. Good crawlers follow it; bad ones ignore it. Even Cloudflare's docs say compliance is voluntary.
  2. *A named group replaces the ` group.** If you write a User-agent: GPTBot` section, GPTBot follows only that section and ignores your general rules. Repeat anything important.
  3. Every subdomain needs its own file. shop.yoursite.com does not inherit from yoursite.com.

The crawler names that matter

Each name below was checked on the company's own page on 9 November 2026. Most companies now run separate bots for training and for search.

  • OpenAI: GPTBot is for training. OAI-SearchBot puts sites into ChatGPT search, and OpenAI says opted-out sites "will not be shown in ChatGPT search answers". ChatGPT-User fetches a page when a person asks, and OpenAI says robots.txt rules may not apply to it. (OpenAI)
  • Anthropic: ClaudeBot collects training data. Claude-SearchBot indexes for search. Claude-User fetches pages for a user's question. (Anthropic)
  • Google: Google-Extended is a control token, not a separate crawler. It covers Gemini training and grounding in Gemini Apps, and Google says it does not affect inclusion or ranking in Google Search. AI Overviews are part of Search, governed by Googlebot. (Google)
  • Perplexity: PerplexityBot is for Perplexity's search results and, per Perplexity, not for training foundation models. Perplexity-User fetches on request and "generally ignores robots.txt rules". (Perplexity)
  • Apple: Applebot-Extended does not crawl; it tells Apple not to train on what Applebot collected. Pages that disallow it can still appear in Siri and Spotlight results. (Apple)
  • Meta: meta-externalagent crawls for AI training. meta-webindexer helps Meta AI cite content. meta-externalfetcher handles user requests and may bypass robots.txt. (Meta)
  • Common Crawl: CCBot builds Common Crawl's open web archive, which anyone, AI developers included, can download and analyse. (Common Crawl)

Blocking training and disappearing from AI answers are two different choices. Mixing them up is an easy mistake, and an expensive one.

Think of a CA firm in Ranchi with clear GST explainers. When someone asks ChatGPT or Perplexity "who does GST filing near me", the answer is built from pages the search bots could read. Block those bots and you remove yourself from the shortlist. Block only the training bots and you still appear, while saying no to being folded into the next model.

Three robots.txt strategies for a small business site
Allow everythingBlock training onlyBlock training and AI search
Google Search and AI OverviewsYesYesYes
Cited in ChatGPT, Claude, Perplexity searchYesYesMostly no (user-triggered fetches may still happen)
Gemini app groundingYesNo, if you block Google-ExtendedNo
Used to train future modelsPossiblyNot by bots that honour the rulesNot by bots that honour the rules
Best forLocal businesses chasing reachMost businesses with original contentPaid, premium or sensitive content

As of 9 Nov 2026. Vendor behaviour and the shared token's rollout are changing; check each company's crawler page.

Copy-paste robots.txt examples

Swap in your own domain, then open yoursite.com/robots.txt to check it.

1. Block training, stay visible in AI search (our default suggestion)

# Shared AI-training opt-out (draft; support rolling out 2026-2027)
User-Agent: __usage_ai-training
Disallow: /

# Vendor training controls that work today
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: CCBot
Disallow: /

# Everyone else, including search and AI-search bots
User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

If Gemini app visibility matters more to you than Gemini training, drop the Google-Extended group; it covers both.

User-Agent: __usage_ai-training
Disallow: /

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Stacking several User-agent lines over one rule set is valid. Some user-triggered fetchers say they may not follow it.

3. Opt out of training except for one folder

This is the pattern from the draft itself: say no to training everywhere except your press kit or public guides.

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Beyond robots.txt: Cloudflare-style controls

If your site sits behind Cloudflare, you get two extra levers. Its managed robots.txt setting can write AI rules for you, including a Content-signal line with search, ai-input and ai-train preferences, and Cloudflare says it is available on all plans. Its AI Crawl Control tools can then actually block named AI bots at the network edge, instead of politely asking. A sign on the door versus a guard at the door. Other CDNs and hosts offer similar rules.

What about llms.txt?

llms.txt is a Markdown file at your site root that gives AI tools a short summary of your site and links to its most useful pages. Its own site calls it "a proposal". It is not an IETF standard, none of the opt-out controls above depend on it, and it does not block anything. It is cheap to add if you want AI tools to understand your services, but it is a signpost, not a fence.

What we know vs what we don't

We know: the token's syntax, that four major labs have publicly backed it, and the dates Microsoft and Google have given.

We don't know: when OpenAI's support goes live, whether smaller crawlers will adopt it, and whether the draft becomes a formal standard. Nothing here removes content already used for training.

Should a small business block AI crawlers?

  • Yes, block everything AI — Fair if your content is paid or sensitive. Just know you will also drop out of most AI search answers, so keep Googlebot allowed and watch enquiries.
  • Block training, allow AI search — The balanced choice for most businesses with original content. Use example 1 above and revisit it in early 2027 when Google and Microsoft switch the token on.
  • No, I want every bit of reach — Reasonable for local service businesses whose pages are basically adverts. Make sure your services, location and contact details are clear so AI answers describe you correctly.
  • I have never looked at my robots.txt — Open yoursite.com/robots.txt in a browser right now. If it is missing or was set by a theme years ago, it is worth ten minutes this week.

Frequently asked questions

Does the new shared AI-training opt-out work today?

Partly. Anthropic says it has begun rolling it out, OpenAI says it will follow within months, Microsoft starts on 1 February 2027 and Google says early 2027. Until then, keep vendor rules like GPTBot and Google-Extended in place.

No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, and only blocking that bot removes you from ChatGPT search answers.

Does blocking Google-Extended hurt my Google ranking?

Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It does, however, cover grounding in the Gemini app, so blocking it can reduce your visibility there.

Can robots.txt remove my content from models already trained?

No. robots.txt and the new token only express preferences for future crawling and training. They do not reach back into existing datasets or models.

Is llms.txt a standard I need?

No. It is a community proposal that helps AI tools find a summary of your site. It is optional and blocks nothing.

The bottom line

The shared token is good news: one line instead of a dozen, backed by the four biggest names. But for the next few months the safe setup is belt and braces, the token plus the vendor rules, with search bots left open unless you have a reason to hide.

If you are still deciding how AI should find your business, start with the basics in our SEO guide for small business websites, check that you actually control your domain and hosting, and see which assistants your customers use in ChatGPT vs Gemini vs Claude. If you would rather have someone set up your robots.txt, bot rules and an AI-friendly site structure properly, talk to us.

Sources and further reading

  1. Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00) · IETF Datatracker · 2026-10-30
  2. Supporting a shared AI training control for websites · Google Search Central Blog · 2026-10-30
  3. Your content, your choice: clearer preferences for training and grounding · Microsoft Bing Blogs · 2026-10-30
  4. Google, Microsoft & OpenAI Plan To Honor Shared AI Training Opt-Out · Search Engine Journal · 2026-11-01
  5. Overview of OpenAI Crawlers · OpenAI · 2026-11-09
  6. Does Anthropic crawl data from the web, and how can site owners block the crawler? · Claude Help Center · 2026-11-09
  7. Google's common crawlers · Google for Developers · 2026-11-09
  8. Perplexity Crawlers · Perplexity · 2026-11-09
  9. About Applebot · Apple Support · 2026-11-09
  10. Meta Web Crawlers · Meta for Developers · 2026-11-09
  11. Common Crawl - CCBot · Common Crawl · 2026-11-09
  12. robots.txt setting · Cloudflare Docs · 2026-11-09
  13. The /llms.txt file, v2 · llmstxt.org · 2026-08-10

Contact Acquiretech Software Solution: info@acquiretechsoftwaresolution.in · WhatsApp +91 80786 33912 · Send an enquiry