X-Robots-Tag on 27 homepages: 1 sends it, and 20 send nothing at all

FIELD TEST · 2026-08-17 · 30 sites · single snapshot

Sample and method: the same 30 sites from our robots.txt and sitemap surveys. One homepage request each, browser user agent, response headers and HTML captured together, 2026-08-17. Three refused us, so counts are out of 27.

The X-Robots-Tag response header is the one place you can set indexing and snippet rules on files that have no HTML to put a meta tag in. Exactly one of 27 homepages sends it. Twenty send neither the header nor a meta robots tag. And of the two sites that do set a snippet rule, both set it to unlimited — they are opening the tap, not closing it.

What 27 homepages actually send

The headline is how empty this layer is. Most sites leave every robots directive at its default and never state anything at all.

How we measured it

The rule was fixed before the crawl. We captured the final response headers after redirects, plus the delivered HTML, in the same request.

1. Read the x-robots-tag header from the last response in the redirect chain.
2. Read <meta name="robots"> from the delivered HTML.
3. Mark a site as setting a snippet rule if either string contains nosnippet or max-snippet.

Homepages only. A site can send X-Robots-Tag on its PDFs and images and nothing on its homepage, and this method would score it as absent.

Every directive we found, verbatim

Seven sites carry a meta robots tag. Here is every value, unedited.

Four of the seven only raise the image preview size. Two say index, follow, which is what happens anyway when you say nothing. That leaves the publishers.

The two sites setting snippet rules are both widening them

Ars Technica and TechCrunch set max-snippet:-1. The value -1 means no limit — take as much text as you want. It is the opposite of the restrictive reading that max-snippet usually gets in SEO writing.

That matters because max-snippet is the directive that also governs how much of a page can be shown in Google's AI surfaces. Two ad-funded publishers, both of whom lose a click when a machine answers instead of them, have explicitly told engines to take as much as they like. We did not ask them why, and we are not going to guess in print.

BBC is the only site using the header, and its meta tag is fossilised

BBC sends X-Robots-Tag: bingbot: noarchive — a per-crawler directive telling Bing not to keep a cached copy. It is the only response-header directive in the whole sample.

Its meta robots tag says noodp, noydir. Those two told search engines not to use descriptions from the Open Directory Project and the Yahoo Directory. Both directories have been shut for well over a decade, and both directives were retired with them. The tag is inert.

We are not calling that a mistake — inert directives cost nothing and get left behind everywhere. It is a good illustration of what an unaudited head section accumulates.

What we cannot tell you

We do not know whether any of this changed an engine's behaviour. We read what sites declare; we have no view of what Google, Bing or any AI engine did with it. We also did not check non-HTML files, which is where X-Robots-Tag earns its keep — a PDF or an image cannot carry a meta tag, and the header is the only lever available.

So the honest version of the headline is narrower than it sounds: one in 27 uses the header on its homepage. Usage on other file types could be much higher and this crawl would never see it.

When you actually need X-Robots-Tag

The header is not a better meta tag. It is the only option for a whole class of files, and using it anywhere else mostly adds a place to forget about.

• Use X-Robots-Tag for files with no HTML head — PDFs, images, plain text, generated exports.
• Use the meta tag for normal pages. It is easier to inspect and easier for anyone else on your team to find.
• Do not write index, follow expecting it to do something. It is the default, and two sites here are shipping it as decoration.
• Do not assume max-snippet restricts anything. -1 means unlimited, and that is what both publishers here chose.

The rule behind all of this — which directive actually governs what an AI answer may quote — is covered in nosnippet is the real AI lever. If you are still deciding which crawlers to admit in the first place, that starts at robots.txt for AI crawlers. Auditing these headers across a whole site, including the file types a homepage crawl never touches, is what QueryWin is being built to do.

Common questions

Is X-Robots-Tag stronger than the meta tag?

Neither overrides the other by rank. They are two delivery mechanisms for the same directives, and the header is the only one available for non-HTML files.

Does max-snippet:-1 do anything at all?

It states a choice explicitly rather than leaving it to the engine's default. Whether the outcome differs from silence is not something this crawl can answer.

Should I add noarchive like BBC?

Only if a cached copy of your page is a problem for you. It is a narrow tool, and 26 of 27 sites here decided they did not need it.

Why do so many sites set max-image-preview:large?

Four of seven do. It allows a bigger image in results, which is a visibility gain with no obvious cost — which is probably why it is the one directive that spread.