The Google-Extended opt out costs you nothing in Google Search

RULE CHANGE · 2026-08-15 · Google and Apple documentation

The Google-Extended opt out costs you nothing in Google Search. Google's own crawler documentation states it plainly, and Apple says the same about its equivalent. Both control tokens govern model training only. The settings that genuinely decide whether you appear in AI answers are different lines entirely, and blocking the wrong one is the expensive mistake.

What the Google-Extended opt out actually does

Two sentences settle most of the argument. Google, on Google-Extended: it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." Apple, on Applebot-Extended: it "does not crawl webpages", and "webpages that disallow Applebot-Extended can still be included in search results."

Both pages were read on 15 August 2026: Google's common crawlers reference and Apple's Applebot page. Neither statement is new. It just does not travel as well as the fear does.

Neither of these is a crawler

This is the part that trips people up. Google-Extended and Applebot-Extended never send a request to your server. Google's documentation says Google-Extended "doesn't have a separate HTTP request user agent string". They are labels that tell a company how it may use content some other crawler already fetched.

So there is no traffic to watch, no status code to read, and no way to confirm from your side that the setting is being honoured. You are taking the operator at its word. That is worth saying out loud, because every other check in this field produces evidence and this one produces none.

What the documentation does not say

It is worth being precise about the edge of the claim, because this is where confident writing usually outruns the source. Google scopes the token two ways: it governs whether crawled content may train future Gemini models, and it does not affect Search inclusion or ranking. That is the whole documented scope.

What the page does not do is enumerate every AI surface Google operates and rule each one in or out. So if your real question is whether this setting changes how often you turn up in AI Overviews, the honest answer is that the documentation does not address it and we have no measurement of our own. What is documented is that AI Overviews are built on Google Search, and Googlebot is the crawler that feeds Search — which is a different line in your file, and one this setting does not touch.

Which line costs you what

Opting out of AI is not one decision, it is two, and they point in opposite directions. Training tokens cost you nothing visible today. Search tokens decide whether an AI assistant can cite you at all.

The bottom four rows are where the damage lives. Someone who wants out of AI training and reaches for a blanket rule can remove themselves from every AI answer surface at once, having intended something much narrower.

The mistake this causes

It usually looks like a single tidy line, written by someone who had read a headline about AI scraping and wanted to be done with it:

User-agent: *
Disallow: /

That does not opt you out of AI training. It opts you out of the internet. And because a crawler obeys only the most specific group naming it, adding a named group elsewhere in the same file does not soften it — the rules do not combine. We wrote up that trap separately in robots.txt for AI crawlers.

What to do today

Decide the two questions separately, in this order, because only the second one is reversible in its effects.

1. Do you want to be cited by AI assistants? If yes, leave OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and bingbot alone. This is the decision that moves traffic.
2. Do you want to be used for model training? If not, disallow GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot. Nothing in the first group changes.

Then confirm the first group still gets through, because a robots.txt edit cannot tell you whether your CDN agrees with it.

Common questions

How did you verify this?

By reading the operators' own documentation on 15 August 2026 and quoting it directly, rather than citing anyone's summary of it. Both quoted pages are linked above. We did not test these tokens on live sites, because they cannot be tested — see the second section.

Does opting out remove content already used for training?

No. These directives are forward-looking. They govern future use, not data collected before the line existed.

Will blocking Google-Extended drop me out of AI Overviews?

We do not have a first-party measurement of this and will not pretend otherwise. What is documented is narrower: Google states the token governs Gemini model training and does not affect Search inclusion or ranking. AI Overviews are built on Google Search, and Googlebot is the crawler that feeds it.

Is there any way to check the setting took effect?

No. That is the honest answer. No request is sent, so nothing appears in your logs either way.

The short version

Two of the scariest-sounding lines in robots.txt cost the least. The lines that actually cost you are the ones nobody warns you about, because they are named after search rather than after AI. Getting a change written, published and pushed for indexing is the part that decides whether any of this shows up.