Meta robots tag: 7 of 28 homepages ship one, and two of the directives died a decade ago
A meta robots tag survey across 30 homepages: 7 of 28 declare one, none uses noindex, three spend it on a default that Google says has no effect, and only two control the snippet length that AI answers read.
FIELD TEST · 2026-08-19 · 30 sites · single snapshot
Sample and method: the same 30 sites used in our robots.txt, sitemap, homepage structure and request chain surveys. One homepage request each, browser user agent, redirects followed, 2026-08-19, from an exit in Japan. Two sites refused us, so counts are out of 28.
Seven of 28 homepages ship a meta robots tag. None of the seven contains noindex, which is expected. What is less expected is that three of them spend the tag on index, follow, which Google's own documentation describes as having no effect, and one ships two directives that have been dead for roughly a decade.
How we measured it
Fixed before the crawl, unchanged afterwards.
- Request the homepage once with a desktop browser user agent, following redirects.
- Read every
<meta>element in the delivered<head>whosenameisrobots,googlebot,google,bingbotorgoogle-extended. - Record the
contentstring exactly as written, including spacing and case.
No rendering. A tag injected by JavaScript would not be counted here, and on a homepage that is a real possibility.
All seven declarations, in full
This is every robots directive we found on the sample, verbatim.
| Site | content value |
|---|---|
| techcrunch.com | index, follow, max-image-preview:large, max-snippet:-1, max-video-preview:-1 |
| arstechnica.com | max-snippet:-1,max-image-preview:large,max-video-preview:-1 |
| www.wired.com | index, follow, max-image-preview:large |
| vercel.com | index, max-image-preview:large |
| www.framer.com | max-image-preview:large |
| www.theverge.com | index,follow |
| www.bbc.com | NOODP, NOYDIR |
The remaining 21 homepages declare nothing at all, which is the correct default for a page you want indexed. Absence here is not neglect.
Half of what is written does nothing
Google's list of valid indexing and serving rules contains twelve entries: all, noindex, nofollow, none, nosnippet, indexifembedded, max-snippet, max-image-preview, max-video-preview, notranslate, noimageindex and unavailable_after. That list, and every Google sentence quoted below, was read on 2026-08-19 on the robots meta tag specification at developers.google.com.
index and follow are not on it. The equivalent is all, and Google's description of it is unusually direct: "There are no restrictions for indexing or serving. This rule is the default value and has no effect if explicitly listed." Four of the seven sites write index anyway, three of them with follow alongside.
The BBC's pair goes further back. NOODP and NOYDIR told search engines not to use descriptions from the Open Directory Project and the Yahoo Directory, and neither directory exists any more. Neither token appears anywhere in Google's current documentation. That tag is a fossil, sitting on one of the most-crawled homepages in the world.
Writing a default into a directive does not make it stronger. It just makes the tag longer.
Two sites control the one directive that reaches AI answers
max-snippet:-1 appears on exactly two homepages in the sample: arstechnica.com and techcrunch.com. Both are publishers, and both are opening the limit rather than closing it — -1 means no limit.
This is the directive that matters most for anyone thinking about AI citation, and Google says so in the definition itself. max-snippet "applies to all forms of search results (such as Google web search, Google Images, Discover, Assistant, AI Overviews, AI Mode) and will also limit how much of the content may be used as a direct input for AI Overviews and AI Mode."
So the count that matters is not seven of 28. It is two of 28: the number of homepages here making any deliberate statement about how much of their content an AI answer may use. Twenty-six leave it to Google.
There is a trap next to it worth naming. Google states that "in the case of conflicting robots rules, the more restrictive rule applies. For example, if a page has both max-snippet:50 and nosnippet rules, the nosnippet rule will apply." A generous value in one place does not survive a restrictive one somewhere else — including in an HTTP header, which is where we found almost nothing last time.
One site uses a directive we could not find documented
Wired ships a second tag: <meta name="google" content="nopagereadaloud">. It is not on the robots meta tag reference page, whose valid-rule list we quoted above, and we did not chase down where else Google documents it or whether it is still honoured. We are recording it because it was there, not because we can tell you what it does today.
We cannot tell you, for any of these seven, whether the value was a decision or a platform default. max-image-preview:large appears on five of the seven, which is the kind of uniformity that usually means a content management system wrote it. From outside the site there is no way to separate a policy from an inherited default.
What to do about your own meta robots tag
Two habits worth keeping, two worth dropping.
- Ship nothing unless you are restricting something. The default is already what most pages want, and 21 of these 28 sites take that route.
- Decide about
max-snippetdeliberately, because it is the one directive here that governs how much of your text an AI answer may use. - Do not write
index, follow. Google's documentation says the default value has no effect if explicitly listed. - Do not leave directives in place because they were inherited. NOODP outlived the directory it referred to by nearly ten years on one of these pages.
Check the header as well as the HTML, because the two can disagree and the more restrictive one wins. Whether a crawler can reach and read the page carrying any of this is a separate question, and the AI crawler accessibility checker is for that one.
How rarely the same directives appear in a response header instead is in the X-Robots-Tag survey, where only one of 27 sites sent one. Why snippet length, and not any imagined opt-out token, is the actual control over AI citation is in there is no noai directive.
Common questions
How did you measure this?
One homepage request per site on 2026-08-19, desktop browser user agent, redirects followed, exit in Japan. We read every robots-family meta element from the delivered head and recorded its content string verbatim. No rendering, so client-side tags are missed.
Do I need a meta robots tag?
Only to restrict something. Twenty-one of these 28 homepages ship none, and that is the right default for a page you want in search results.
Is index, follow useful?
No. Google documents all as the default value that has no effect if explicitly listed, and index and follow are not in its list of valid rules at all.
Which directive affects AI Overviews?
max-snippet and nosnippet. Google's definitions say both apply to AI Overviews and AI Mode and limit how much of the page may be used as a direct input.
What happens if the header and the HTML disagree?
The more restrictive rule applies. Google gives the example directly: a page with both max-snippet:50 and nosnippet gets nosnippet.
