Naming a crawler usually means blocking it: 82 of 99 groups

FIELD TEST · 2026-08-15 · 14 sites · 99 named groups

Sample and method: 14 well-known sites that name AI crawlers in robots.txt. We fetched each file with the OAI-SearchBot user agent on 15 August 2026, parsed the groups, and checked whether each named crawler got a full-site Disallow: / or only path-level rules.

Naming a crawler usually means blocking it: 82 of 99 named groups (82%) were a full-site Disallow: /. But the exception is not noise — three sites named crawlers and blocked none of them, using the groups for path rules instead. Counting names, which is what most surveys do, conflates two opposite intentions.

How we measured it

We parsed each robots.txt into groups, treating consecutive User-agent: lines as sharing the rules beneath them, then asked one question per named AI crawler: does this group contain Disallow: / with nothing else permitting it back in?

This is deliberately crude. It does not model Allow overrides or wildcard precedence, so a group that blocks everything then re-permits a subtree counts here as a full block. That would understate openness, not overstate it.

Naming a crawler usually means blocking it: site by site

Ordered by how many AI crawlers each file names. The last three rows are the exception.

Two populations, not one distribution

The 82% figure is an average over two groups that behave nothing alike. Ten sites blocked essentially everything they named. Three named crawlers and blocked none of them — netlify.com, cloudflare.com and supabase.com all used named groups purely to scope which paths a crawler may take.

So a survey that counts "sites naming AI crawlers" is measuring two opposite decisions with one number. A site that names five crawlers to keep them out of /admin/ is doing the reverse of a site that names five to keep them out entirely.

What this means for reading anyone's robots.txt

Including your competitors'. The presence of a named group tells you someone thought about it. Only the rules underneath tell you what they decided, and those are two lines further down than most people look.

The same applies to your own file after a template or platform default has touched it: a named group you did not write may be permitting rather than restricting, or the other way round.

What this does not show

Fourteen sites, one snapshot, and a parser that does not implement the full precedence rules of the standard. It also cannot see intent — a full block might be a deliberate policy or a copied template, and from outside those look identical.

We also did not check whether any crawler honours these files. That would need server logs, which we do not have for anyone else's site.

Common questions

How did you measure this?

One curl per site, a small parser that groups consecutive User-agent: lines with the rules beneath them, then a check for Disallow: / per named crawler. The raw output is saved alongside this post.

Why does a full block versus a path rule matter so much?

Because one removes you from an answer engine entirely and the other does not. If you are copying a configuration from a site you admire, that difference is the whole configuration.

Is 82% high or low?

It is high, and it is also an artefact of which sites name crawlers at all. Publishers name them and block them; most other sites do not name them. The 82% describes the naming population, not the web.

The one number to take away

Of 99 named AI crawler groups, 82 were a full-site block — but three of the fourteen sites blocked nothing they named. Before copying anyone's robots.txt, read two lines further down, then ship your own version and push it for indexing. The companion breakdown of who names what is in who blocks AI crawlers and who publishes llms.txt.