Who blocks AI crawlers and who publishes llms.txt: 30 sites, two opposite strategies

FIELD TEST · 2026-08-15 · 30 sites · single snapshot

Sample and method: 30 well-known public sites. We fetched /robots.txt and /llms.txt from each with the OAI-SearchBot user agent on 15 August 2026, and counted which of 16 named AI crawlers appear as their own User-agent: group.

Who blocks AI crawlers turns out to depend almost entirely on what the site sells, and two groups are doing exactly opposite things. News and publishing sites name AI crawlers by the dozen in robots.txt and publish no llms.txt at all. Developer-tool and SaaS companies publish llms.txt and name zero AI crawlers. Across 29 readable robots.txt files, 13 named at least one AI crawler; 14 of 30 sites served a real llms.txt.

How we measured it

One request per file per site, no rendering, no retries. A site counts as naming a crawler only when that crawler has its own User-agent: line — we did not try to interpret what the rules underneath it allow or forbid — see robots.txt for AI crawlers for why a named group changes more than it looks, because the interesting signal here is whether anyone bothered to name it at all.

For llms.txt we required a 200 and a body that is not HTML, because a single-page app will happily return its own homepage for any path. In this sample that particular trap never fired: every 200 we got was a real text file.

Who blocks AI crawlers: the publishers

Every site in this group names a large number of AI crawlers and none of them publishes an llms.txt.

Sixteen named crawlers is not a policy anyone arrives at by accident. It is a maintained list, updated as new crawlers appear, which means somebody at these organisations owns this file.

The developer-tool companies

The same measurement, the opposite result.

Ten sites, zero named AI crawlers between them, and an llms.txt on every one. A handful of others sit in the middle: netlify.com names 6 and publishes one, cloudflare.com names 5 and publishes one, supabase.com names 4 and publishes one, notion.com names 3 and publishes one.

Why the split makes sense

It follows from what each business sells. A publisher's product is the article, so a model trained on the article competes with it. A developer-tool company's product is the tool, and being quoted accurately in an AI answer is free distribution — the documentation is marketing, not inventory.

Which means the common advice to "block AI crawlers" is not advice at all. It is one industry's answer being repeated to everyone.

What we did not find

No site in this sample did both aggressively — nobody published a detailed llms.txt while also blocking a long list of crawlers. We expected a few, since the two files govern different things and combining them is perfectly coherent. Thirty sites is far too small to conclude that nobody does it.

We also cannot see intent. A site with zero named AI crawlers may have decided to allow everything, or may never have thought about it. Those two look identical from outside, and this measurement cannot separate them — which is the biggest limitation of the whole exercise.

The most-named crawlers

Across the 29 readable files, these appeared most often as their own group.

The training crawlers lead. That is consistent with the publisher motivation above, and it means most of these files were written to answer the training question rather than the visibility one.

One site answered with a joke

stackoverflow.com returned 418 to our request for robots.txt. That status code is the April Fools' "I'm a teapot" from 1998, and using it here is a deliberate refusal with a wink. We could not read their file, so they are excluded from the 29.

Common questions

How did you measure this?

Two GET requests per site with the OAI-SearchBot user agent, then a regex for each of 16 crawler names as its own User-agent: line. Raw results are in the JSON we saved alongside this post. Anyone can re-run it; the method is two lines of shell per site.

Does naming a crawler mean blocking it?

No, and we deliberately did not measure that. A named group can allow, restrict a path, or set a crawl delay. We counted attention, not permission.

Should I copy what the publishers do?

Only if you sell what they sell. If your pages exist to bring people to a product, the publisher configuration removes you from the answers where those people are now asking their questions.

The one line to take away

Thirty well-known sites, two coherent and opposite strategies, and no overlap between them. Before copying either, work out which product you have. If your content is the thing people pay for, the publishers are your reference; if your content is how people find the thing they pay for, blocking is the expensive option — and getting that decision shipped, published and pushed for indexing is the part that actually changes anything.