robots.txt for AI crawlers: the rule that breaks most files

A robots.txt for AI crawlers is one plain text file that names each crawler and states what it may fetch. The syntax is not what breaks it. This is: a crawler that finds its own name in your file stops reading your User-agent: * rules entirely. Add one line for GPTBot and every rule you wrote for everyone else can quietly stop applying to it.

Read this first

This chapter assumes AI crawlers can already reach your server. If you have not confirmed that, do it before you edit anything — a file full of Allow lines proves nothing when the refusal happens one layer above your site. The two-minute version is in how to check if AI can read your site.

What a robots.txt for AI crawlers can and cannot do

It declares intent. It is a file crawlers choose to read and obey, and it enforces nothing on its own. Well-behaved AI crawlers honour it. Your CDN, your firewall and your origin server are what actually decide whether bytes come back, and none of them read this file.

robots.txt decides what you permit. It never decides what you serve.

The rule that breaks most robots.txt files

A crawler obeys exactly one group of rules — the group naming it most specifically — and it does not fall back to your general rules for anything the specific group left out. Google's specification is blunt about it: "Only one group is valid for a particular crawler", and "User agent specific groups and global groups (*) are not combined."

Which means this file does not do what it looks like it does:

User-agent: *
Disallow: /admin/
Disallow: /cart/

User-agent: GPTBot
Disallow: /drafts/

Read it as a person and GPTBot is kept out of three directories. Read it as GPTBot and only one line exists: /drafts/. Your admin and cart paths went back on the menu the moment you typed that second group. Nothing warns you, and the file still validates.

The fix is to repeat every shared rule inside each named group. Repetition looks wrong and is correct.

User-agent: GPTBot
Disallow: /admin/
Disallow: /cart/
Disallow: /drafts/

Sort the crawlers into three tiers before you write a line

The decision is not per crawler, it is per job. Three jobs, three answers. Search crawlers feed the answers people see this week, so blocking them removes you from those answers. Training crawlers feed models shipped later. User-triggered fetchers run because a person asked a question right now.

Tokens and purposes come from the operators themselves: OpenAI, Perplexity, Anthropic, Google, Apple, Meta and Common Crawl. Check them before you copy anything, including the block below — this list moves.

A robots.txt for AI crawlers you can paste

This is the allow-search, decide-training shape. Replace the two private paths with your own, and delete the training group if you would rather be trained on.

# Search and answer crawlers — allowed, minus private paths
User-agent: Googlebot
User-agent: bingbot
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
User-agent: Meta-WebIndexer
Disallow: /admin/
Disallow: /cart/

# Model training — refused. Delete this group to allow it.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Everyone else
User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml

Stacked User-agent lines share one set of rules, so each block above is a single group, not six. Note that the private paths are written twice on purpose — that is the repetition the previous section demands.

Two lines in that file are not crawlers

Google-Extended and Applebot-Extended never send a request to your server. They are control tokens: they change how content that another crawler already collected may be used. There is no traffic to observe and no live test that can confirm your setting took effect.

Both are widely misread as ranking risks, and both operators say otherwise in writing. Google: Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search". Apple: Applebot-Extended "does not crawl webpages", and "webpages that disallow Applebot-Extended can still be included in search results".

What robots.txt will not stop

Fetchers that run because a user asked something are a separate category, and at least one operator states plainly that it skips your file. Perplexity's documentation on Perplexity-User: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules."

The honest position is that the bottom tier of the table is not under your control through this file. If a page must not be fetched on demand, robots.txt is the wrong tool — authentication is the right one. We do not have measurements on how often each of these fetchers actually arrives, and we would not trust a number we had not taken from our own logs.

Three ways this goes wrong

All three are silent. The file stays valid, no tool reports an error, and the damage only shows up later as traffic that never arrives.

1. Blocking GPTBot and expecting ChatGPT to disappear. GPTBot is the training crawler. What decides whether ChatGPT can surface you today is OAI-SearchBot. Blocking the first while leaving the second open is a coherent choice; confusing them is not.
2. Adding a named group and losing the shared rules. The failure from the third section. After every edit, re-read the file once per named crawler and ask what that crawler alone can see.
3. Editing the file on the wrong host. Rules apply only to the host, protocol and port serving them, so https://example.com/robots.txt governs nothing on https://www.example.com. If both hostnames answer, both need a file.

Common questions

Does blocking Google-Extended hurt my Google rankings?

No, and Google says so directly: it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search". It governs Gemini model training only.

Does adding a Disallow remove content already used for training?

No. Every one of these directives is forward-looking. It governs future fetches, not data collected before you wrote the line.

How often should I revisit the file?

When a new crawler appears — Meta-WebIndexer is recent enough that most published examples predate it — and after any platform migration, because a new host ships its own default file.

Next step

Your file now says what you mean. Whether your server agrees is a different question, and the usual disagreement is a CDN refusing crawlers your robots.txt welcomes. That is the next chapter, and it is the most common single reason a site with a perfect file is still invisible.

Editing one file is the cheap half. Getting a change written, published and pushed for indexing is where most sites stall.