Block AI training bots without losing search: what Cloudflare changed on September 15

RULE CHANGE · 2026-09-15 · Cloudflare

Cloudflare changed what its Block setting does on 15 September 2026. If you were using it to block AI training bots, it now also stops Googlebot, Bingbot and Applebot — search included. The replacement is a setting called Disallow AI Training, and all it does is publish a no-training preference in your robots.txt. Search keeps working.

What changed on September 15

Cloudflare announced a new control and a new designation on 15 September 2026. The control, Disallow AI Training, writes a no-training preference into your robots.txt. The designation, Accountable, is a label Cloudflare gives bot operators that meet four requirements: a robots.txt opt-out for training, an opt-out for AI summaries, URL-level visibility into which pages were made available for training, and an assurance that opting out of training will not affect traditional search results. Apple, Google and Microsoft are listed as meeting it.

The part that will bite people is the redefinition. Cloudflare's announcement states that Block and Block on pages with ads "now apply to mixed-use crawlers, including Applebot, Bingbot, and Googlebot, so either setting impacts search as well as training." Before this change those two settings deliberately skipped mixed-use crawlers for exactly that reason. If your setting has said Block since 2024, you may have been thinking it meant "no AI" while it now means "no search either". The announcement's own instruction is blunt: "If you want mixed-use crawlers gone entirely, you now have to say so. Select Block."

Three other things changed in the same release. The single blanket toggle that used to cover all of this is deprecated in favour of separate Search, Training and Agent controls. Managed Robots.txt is deprecated in favour of Bot Preference Sync. And Disallow AI Training becomes part of the recommended configuration offered to new domains, with the more restrictive preset going to sites that earn money from ads.

Source: Cloudflare, Have it both ways: stay discoverable in search while disallowing AI training, published 15 September 2026, read on 21 September 2026.

The distinction this turns on

A mixed-use crawler is one user agent doing two jobs: building a search index and collecting training data. Googlebot, Bingbot and Applebot are all mixed-use. That is why blocking them was historically a package deal, and why Cloudflare could not offer a "no training" switch for them until the operators agreed to honour one.

The distinction that matters for you is between a crawler and a control token. A crawler sends requests to your server. A control token does not — it is a name in robots.txt that tells a company how it may use content some other crawler already fetched. Google's crawler documentation is explicit about its own token: "Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity."

That has a practical consequence. When you disallow a control token, nothing appears in your logs, so you cannot confirm from your side that the preference was honoured. You are taking the operator at its word. Everything else in this field produces evidence; this one does not.

What each operator actually gives you

Four operators cover the mixed-use crawlers you are most likely to care about. Each has published its own mechanism, and none of them requires a third party to be involved. The table below is built from the operators' own documentation, not from summaries of it.

Google's wording is worth keeping: Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." Apple's is the same in substance — Applebot-Extended "does not crawl webpages", and "webpages that disallow Applebot-Extended can still be included in search results." Microsoft committed to the same thing in 2023: content tagged NOARCHIVE "will not be used for training Microsoft's generative AI foundation models", and content with the NOCACHE or NOARCHIVE tag "will still appear in our search results".

One gap is worth naming. Cloudflare's post says Microsoft is still building the mechanism to respect a no-training preference in robots.txt at the domain level, targeted for early 2027. Until that ships, choosing Disallow AI Training in Cloudflare does not by itself convey a no-training preference to Bing. The NOARCHIVE meta tag is the mechanism that works today.

How to block AI training bots without losing search

Treat it as two decisions, not one, and make them in this order. The second one is the only one that can cost you traffic if you get it wrong.

1. Decide whether you want to be cited by AI assistants. If yes, leave the search-side crawlers alone: OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and bingbot. Done when none of those five appears in a Disallow line for a public path.
2. Decide whether you want to be used for training. If not, add the control tokens rather than the crawlers: Google-Extended and Applebot-Extended, plus GPTBot, ClaudeBot, CCBot and meta-externalagent for the operators that keep training on a separate user agent. Done when each token has its own group with a Disallow: /.
3. Check that the two groups do not collide. A crawler obeys only the most specific group naming it, and the rules do not combine. If you already have a named group for Googlebot, adding a second one does not merge with it. Done when each user agent you care about appears exactly once.
4. Re-read the file as a crawler would. Start from the top, find the first group whose user agent matches, and stop there. Whatever is in that group is your whole policy for that crawler.

Step 4 is the one people skip, and it is the step that catches the mistake described below.

We checked our own six sites first

Before writing any of this up, we fetched our own sites' robots.txt files and counted. All six name both control tokens. None of them disallows either token on a public path, which means none of them expresses a training opt-out at all — the tokens are listed, then allowed along with everything else.

Sample and method: six sites we operate, one GET per /robots.txt on 21 September 2026 with a desktop browser user agent, no rendering. We read the file as text and looked for the token name and the directives in its group. Six of six is not a survey, it is a spot check on sites we control — and it produced the least flattering possible result for a site writing about this, which is why it is in here.

There is a second reason to publish it. Cloudflare's post gives its own numbers: fewer than 1% of sites on Cloudflare choose to block search bots, and 17% enable some mechanism to block training. Our six sites are sitting in the 17% that did not. If you are reading this because you assumed your robots.txt already said something about training, the honest position is that most files say nothing at all.

Common questions

How did you verify the Cloudflare changes?

By reading Cloudflare's own announcement, published 15 September 2026, and quoting the parts that change behaviour rather than the parts that describe intent. Every operator claim in the table above was checked against that operator's own documentation or its own blog post, and each is linked in the body. We have not tested the settings inside a Cloudflare account, so we cannot tell you what the dashboard looks like on your plan.

Does Disallow AI Training affect my Google rankings?

No, and Google says so in its own crawler reference: Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". What Cloudflare publishes under Disallow AI Training is a no-training preference, not a no-search preference. The setting that does affect search is Block.

If I already have "Block" selected, am I blocking Googlebot right now?

Possibly, and you should check rather than assume. The announcement says Block now applies to mixed-use crawlers including Googlebot, Bingbot and Applebot. Its migration table shows the legacy Block setting being carried forward as Search: Allow, Training: Disallow AI Training, Agent: Block on pages with ads — so a migrated site keeps search. A site that actively selects Block today, after the change, is asking for search to stop too.

What is the difference between Disallow AI Training and blocking training crawlers directly?

Reach, mostly. Blocking GPTBot or ClaudeBot directly is something you can do in robots.txt with no provider involved, because those operators already run training on a separate user agent. Disallow AI Training exists for the three operators that do not: Apple, Google and Microsoft, whose search and training crawls are the same user agent. If you want to block AI training bots from those three without touching search, you cannot separate the two jobs in your own file — the separation has to come from a control token or a meta tag.

We could not test whether these preferences are honoured. Should we trust them?

You have no way to check, and neither do we. That is the honest answer, and it applies to every training opt-out in this field, including the ones Cloudflare can enforce at the edge and the ones you write yourself. What you can check is the part that is measurable: whether your search-side crawlers still get through. That one leaves log entries.

The short version

Block means something different than it did a week ago, and the new setting you want is Disallow AI Training. To block AI training bots for free, disallow Google-Extended and Applebot-Extended; the lines that actually cost you traffic are named after search. If you want the longer version of that argument, we wrote it up in the Google-Extended opt out, and the crawler-versus-control-token trap is in naming a crawler usually means blocking it. Turning any of it into a change that actually gets picked up is the part QueryWin works on.