Should I block AI crawlers? It is three decisions, not one

Should I block AI crawlers is not one decision. It is three, taken separately: whether to be found in AI search, whether to be used for model training, and whether to serve agents fetching on a user's behalf. Most sites that "blocked AI" made all three at once without noticing, and the one they regret is the first. This chapter is the decision tree, plus what it costs to be wrong in each direction.

Read this first

This is an L3 chapter about a trade-off, not a how-to. The mechanics of writing the rules are in robots.txt for AI crawlers, and which crawler belongs to which surface is in how AI engines pick sources. Read this one to decide what you want; read those to implement it.

One assumption throughout: you are choosing for a site whose content is public. If your material sits behind a login or a paywall, the crawler question is downstream of an access-control question you have already answered.

Split the decision into three

Every vendor now separates these, and the separation is what makes a considered answer possible. Blocking is not a single switch, and treating it as one is how sites end up invisible in AI search while still being trained on by whoever ignores robots.txt.

Notice that only the first row involves a genuine trade. The second is a one-sided giveaway with no observable return, and the third is barely a decision at all — which is why lumping the three together produces such bad outcomes.

The third row is the easiest and the most often got wrong. When a user pastes your URL into a chat and asks about it, a blocked agent means that person is told your page cannot be read. Nobody chose that outcome deliberately.

Should I block AI crawlers: the decision tree

Four questions, in order. The first one that yields a clear answer ends the exercise — this is deliberately not a scoring rubric, because the answers are not additive and pretending otherwise produces a number nobody trusts.

1. Is the content your product? If people pay for the text itself — research, a database, a course — training access gives away the product. Block training, allow search.
2. Does a citation without a click still pay you? Brand-led businesses, tools, local services: being named in an answer is worth something even with no visit. Allow both search and user fetch.
3. Is your revenue per pageview? Ad-funded publishing is the one case where an answered question is a lost transaction. Even then, note what our survey found below before deciding.
4. Can you afford the traffic? A crawl has a real bandwidth cost. If your hosting bills by transfer and your site is large, this is a budget question rather than a strategy one.

Three site types, three different answers

Read the row that matches you, then check it against the tree above. Where the two disagree, the tree wins — it knows something about your business that a category label does not.

The pattern across all three is the same: allowing search is close to free, and the training question is where the actual disagreement lives. That is also why the two settings being independent matters so much — it is the difference between a considered position and a blanket refusal.

What ad-funded publishers actually chose

Row three of the tree is the hard one, so it is worth looking at what publishers did rather than what the argument says they should do. Two of our own surveys touch it.

In our robots.txt survey of 30 sites, publishing sites named AI crawlers far more often than developer-tool sites did, and naming a crawler almost always meant disallowing it — across 99 crawler groups, 82% were full-site disallows. So the industry that loses most from a click-free answer has, broadly, blocked.

But when we looked at snippet directives on 27 homepages, the two sites setting a snippet rule were Ars Technica and TechCrunch, and both set max-snippet:-1 — no limit on how much may be shown. Two ad-funded publishers explicitly widening what engines may quote. We did not ask them why, and we are not going to guess.

Those two findings do not resolve into a single position, and we are reporting that rather than picking the half that makes a tidier argument. The write-ups are naming a crawler usually means blocking it and our survey of X-Robots-Tag across 27 homepages.

The cost side, which we cannot price for you

Crawling consumes bandwidth, and on transfer-billed hosting that is a real line item. We would like to give you a number — AI crawler traffic as a share of requests, converted to money at real billing rates — and we do not have one.

This matters more than it sounds, because the cost argument is the one most often made without evidence. It is the easiest half of the debate to assert and the hardest to check, and a number with no stated source is doing rhetorical work rather than informative work.

Producing it requires server access logs across a period long enough to be representative, plus each host's actual billing terms. That measurement is planned and has not run. Until it does, anyone quoting you a percentage should be asked whose logs it came from and over what window.

What we can tell you is the shape of the check: aggregate your access log by user agent, sum response bytes rather than request counts, and compare against your bill. Response bytes and billed transfer will not match — compression, cache hits and edge behaviour all sit between them — so treat the log figure as an upper bound and not as the invoice.

How reversible is it

Less than you would hope, and this belongs in the decision rather than after it. Undoing a block is one edit, but the edit only takes effect when each crawler next reads your robots.txt, and none of the vendors publish how often that happens.

We have not measured it either — the experiment needs server logs we do not yet have access to. So plan on the change being slow to land and treat a block as a position you hold for months, not as a switch you flip while you think it over.

Three ways this decision goes wrong

None of these announce themselves. Each produces a site that looks configured and behaves differently from what its owner believes, which is why they persist for months.

1. Blocking everything after a news cycle. The cost lands on the search surface, which is the half you wanted, and it lands quietly — nothing reports that you stopped being eligible.
2. Blocking in robots.txt and assuming it is enforced. It is a request. Enforcement lives at the CDN, and the two disagree more often than people expect.
3. Deciding once and never revisiting. Vendors add agents and rename them; a rule set written last year may name nothing that still exists.

What this chapter cannot settle

Whether allowing AI search costs you traffic overall is unresolved, and we have not measured it. The honest position is that citation and click-through are different currencies, and which one matters depends on a business model we do not know.

We also cannot verify vendor claims about how fetched content is used. Perplexity states its search crawler is not used for foundation model training; OpenAI documents its bots as independent settings. Those are statements on the record, not things we can audit, and treating them as either gospel or lies would be equally unfounded.

Keeping a decision like this actually enforced — the same answer at robots.txt, at the CDN and in your headers, across every subdomain — is what QueryWin is being built to check.

Common questions

Should I block AI crawlers if I sell ads?

It is the strongest case for blocking training and the weakest case for blocking search, because a citation still carries your name. Our surveys found publishers broadly blocking and two of them simultaneously allowing unlimited snippets, so the industry has not settled it either.

Does blocking actually stop anything?

It stops crawlers that honour robots.txt. It does nothing to anything that ignores it, which is why enforcement at the network edge is a different and stronger lever.

If I block training, do I lose AI search visibility?

Not where vendors document the settings as independent — OpenAI does, explicitly. You have to name the right agents to get that outcome, and naming the wrong one is the most common way this goes wrong.

Can I allow some crawlers and not others?

Yes, and that is the normal shape of a considered answer. Group by what each agent feeds rather than by which company owns it.

How often should I revisit this?

When a vendor announces a new agent, and otherwise on a fixed schedule you actually keep. A rule set nobody has read in a year is a decision nobody is making.