The GPTBot user agent, and the seven other names in your robots.txt

The GPTBot user agent is one of eight names you can address in robots.txt, and three of the eight are not crawlers at all. They are control tokens: strings that change how your content may be used, sent by no request you will ever see in a log. Getting this distinction right is the difference between a robots.txt that does what you meant and one that quietly does something else.

Read this first

This chapter is the hub for the admission cluster. It covers who to name and what naming each one costs. The syntax of the file itself, including how groups and precedence work, is in robots.txt for AI crawlers. Which crawler feeds which answer surface is in one company, several crawlers.

Everything below assumes your pages are reachable and contain text in the delivered HTML. Admission is worth nothing to a page that arrives empty.

Crawlers and control tokens are not the same thing

A crawler sends HTTP requests with a user agent string. You can see it in a log, you can block it at your CDN, you can verify it by reverse DNS. A control token never sends anything. It exists only as a name you write in robots.txt, and the vendor reads that name to decide what it is allowed to do with content it already fetched under a different agent.

Google states this outright for its own token: "Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity." Apple says the same about its equivalent: "Applebot-Extended does not crawl webpages. Webpages that disallow Applebot-Extended can still be included in search results." Both read 2026-08-18 on the vendors' own documentation.

If a name never appears in your access log, blocking it at the firewall accomplishes nothing. It is a request you make of the vendor, not a door you close.

This is why a CDN rule that blocks "AI traffic" and a robots.txt line that disallows Google-Extended are not two ways of doing the same thing. One drops requests. The other declines a permission.

The eight names worth knowing, and what each one costs

Every user agent string below is copied from the vendor's own documentation, read 2026-08-18. Version numbers change; the token before the slash does not.

Read the third column as a price list, not a warning. Two of these prices are close to zero if you do not care about model training, and three of them cost you presence in an answer surface where people are actively looking for what you sell.

The GPTBot user agent string, and five more

Paste these into a log filter, not into robots.txt. Robots.txt matches on the token, and a longer string there will match nothing.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)

Two gaps in that list are deliberate. Anthropic's support page names ClaudeBot, Claude-User and Claude-SearchBot and explains what each does, but does not publish full user agent strings on that page — so we are not printing one. Google-Extended and Applebot-Extended have no string to print, which is the whole point of the previous section.

How to decide, in four steps

Run this once per site. It takes about twenty minutes and the answer holds until your business model changes.

1. Decide whether you want to appear in AI answers at all. If your content is the product and it sits behind a paywall, the honest answer may be no, and the rest of this changes shape.
2. Allow every search-facing agent: OAI-SearchBot, PerplexityBot, and the two user-triggered fetchers. These are the ones that put a link back to you in front of a reader.
3. Decide about training separately: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended. Blocking all four costs you nothing in any answer surface, and buys you nothing measurable either. It is a position, not an optimisation.
4. Verify with a request, not with the file. Fetch your own homepage sending each user agent above and confirm you get a 200 with body text. Your CDN can override robots.txt, and usually nobody told it what you decided.

Step four is where most sites discover their real configuration. A file that says allow and an edge rule that says deny is not a contradiction the crawler resolves in your favour.

The deliverable: a robots.txt block you can paste

This admits every search-facing agent and declines training use. Change the four disallow groups to allow if you would rather be in the training data.

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

Replace the sitemap line with your own complete URL. Add nothing else: a group that names an agent and then leaves it at the default is a group that says nothing, and it makes the file harder to read six months from now.

Three ways this goes wrong

Each of these produces a file that looks correct and behaves differently from what you intended.

1. Blocking GPTBot to stay out of ChatGPT. Those are different agents, and OpenAI's documentation says the settings are independent. Blocking the training crawler while leaving the search crawler open is a coherent position; assuming one implies the other is not.
2. Naming an agent and blocking it by reflex. Our own measurement of 30 sites found that among 99 named groups, 82 percent were a sitewide disallow — naming a crawler and admitting it is the rarer choice. The details are in naming a crawler usually means blocking it.
3. Writing a full user agent string as the token. Robots.txt matches the product token, case-insensitively, against a substring of the agent. A whole Mozilla string in the User-agent line matches nothing and silently disables the group.

Where this chapter stops

Admission is a permission, not an outcome. Allowing every agent above does not mean any of them will visit, and this chapter cannot tell you whether they did — that needs server logs, which is a separate job. Nor does any of it govern how much of your page may be shown once you are in; that is a snippet directive, and it is a different mechanism entirely.

Vendor documentation also moves. Every string and every claim here was read on 2026-08-18, and the tokens have been stable far longer than the version numbers. Re-read the vendor pages before you rely on a version number for anything.

Auditing this across every site you run, and catching the day an edge rule starts contradicting the file, is what QueryWin is being built to do.

Common questions

What is the GPTBot user agent used for?

OpenAI's documentation says GPTBot "is used to crawl content that may be used in training our generative AI foundation models". It is not the agent that decides whether you appear in ChatGPT's search answers — that one is OAI-SearchBot.

Should I allow AI crawlers at all?

Split the decision. Search-facing agents send readers back to you and cost you a fetch. Training crawlers take content and return nothing measurable. Most sites want the first and can take either position on the second.

Does blocking Google-Extended remove me from AI Overviews?

No. It governs training use for Gemini. Google's AI features in Search are fed by the regular index, so the control that affects them is a snippet directive, not this token.

Do user-triggered fetchers obey robots.txt?

OpenAI states that because ChatGPT-User actions are initiated by a person, robots.txt rules may not apply. Treat those two agents as a courtesy setting rather than a lock.