One company, several crawlers: blocking the wrong one costs you
Every major AI company now runs several crawlers with different jobs. Blocking GPTBot does not remove you from ChatGPT — that is OAI-SearchBot. Here is the map from each operator's own docs.
RULE CHANGE · 2026-08-15 · OpenAI, Anthropic, Meta, Google and Apple documentation
GPTBot vs OAI-SearchBot is the distinction that matters most, and it generalises: every major AI operator now runs several crawlers with different jobs, so blocking the wrong one is the most expensive mistake available. Blocking GPTBot does not remove you from ChatGPT — that is OAI-SearchBot. Meta runs five. Google's AI-related token is not a crawler at all. Here is the current map, taken from each operator's own documentation.
How we checked this
We read each operator's published bot documentation on 15 August 2026 and recorded the robots.txt token and the stated purpose for each. No third-party lists, no summaries — the pages themselves. Every purpose below is the operator's own description, condensed.
GPTBot vs OAI-SearchBot, and the rest of the map
Grouped by operator. The bolded purposes are the ones that decide whether you appear in answers today.
| Company | Token | Job |
|---|---|---|
| OpenAI | GPTBot | Crawls content for training foundation models |
OAI-SearchBot | Powers ChatGPT's search features. Opted-out sites do not appear in ChatGPT search answers | |
ChatGPT-User | User-initiated fetches; not automatic crawling | |
| Anthropic | ClaudeBot | Collects web content that may contribute to training |
Claude-SearchBot | Improves search result quality for users | |
Claude-User | Supports user-initiated requests needing web access | |
| Perplexity | PerplexityBot | Surfaces and links sites in Perplexity results. Explicitly not used for foundation model training |
Perplexity-User | User-triggered fetch. Generally ignores robots.txt | |
| Meta | Meta-WebIndexer | Improves Meta AI search result quality |
Meta-ExternalAgent | Training foundation models, and indexing content | |
Meta-ExternalFetcher | Fetches individual links at a user's request | |
Meta-ExternalAds | Advertising and business products | |
FacebookExternalHit | Crawls content shared on Meta's apps | |
Google-Extended | Gemini training only. Not a crawler — no separate HTTP user agent | |
| Apple | Applebot-Extended | Model training only. Does not crawl webpages |
The three jobs, and only one of them costs you visibility
Sort by job rather than by company and the decision gets much simpler.
| Job | Tokens | Blocking costs |
|---|---|---|
| Search and answers | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer | Your place in those answers. |
| Model training | GPTBot, ClaudeBot, Meta-ExternalAgent, Google-Extended, Applebot-Extended | Nothing visible today. |
| User-triggered | ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher | Not reliably controllable through robots.txt. |
The specific confusion this causes
The most common version: someone reads about AI scraping, adds User-agent: GPTBot and Disallow: /, and believes they have opted out of ChatGPT. They have opted out of training. ChatGPT's search feature runs on OAI-SearchBot, which is still allowed and still indexing them — which is, for most sites, the outcome they actually wanted.
The mirror-image mistake is worse: a blanket rule that catches every token, removing the site from four answer engines to prevent training it did not mind much either way.
Meta is the one to re-check
Meta now documents five separate agents, and Meta-WebIndexer — the one that feeds Meta AI search results — is recent enough that most published robots.txt examples predate it. If you copied a config more than a few months old, it is unlikely to mention it at all.
This is the general shape of the problem. These lists change, and a robots.txt written once is a snapshot of whatever the landscape looked like that afternoon.
What we did not verify
Whether these crawlers behave as documented. We read what each company published; we did not test whether a blocked token stops requesting, and we have no server logs from a site that blocks them. Every purpose above is a claim by its operator, not an observation of ours.
Common questions
How did you verify this?
By reading each operator's own bot documentation on 15 August 2026 and taking tokens and purposes from those pages directly. The sources are linked from robots.txt for AI crawlers, which also covers the group-matching rule that decides which of your rules a given crawler obeys.
Does blocking the training crawlers hurt me?
Not in today's answers, on the operators' own descriptions. It is a business decision about future models, not a visibility decision.
How often does this list change?
Often enough that we would not trust a copy of it, including this one, six months from now. Check the operator pages before you edit.
The one line to take away
One company, several crawlers, different jobs — and only the search ones cost you anything today. Getting that distinction into your robots.txt, and pushing the change for indexing, is a ten-minute job that most sites have never done deliberately.
