robots.txt is not valid: what a strict check of 30 robots.txt files actually flags

FIELD TEST · 2026-09-14 · 30 domains · 30 robots.txt files · one pass

Sample and method: the same 30-domain panel this series has used since 2026-08-15, one request per domain on 2026-09-14 with a desktop Chrome user agent, no JavaScript. We saved each file byte for byte and re-counted every number below from those saved bytes. A file is marked against only the rules Google publishes, never against what other engines accept.

When a validator says robots.txt is not valid, it is almost never reporting a broken rule. It is reporting a line Google never reads. Across 30 robots.txt files and 10,775 rule lines, exactly one file carries an encoding problem, 13 use field names outside the four Google supports, and one answers an error code that makes Google treat the whole file as absent.

How we measured

We fetched each domain's robots.txt once, saved the raw bytes, and then ran two checks: one on the file as a whole, one on each line. The line check fails a file only for something we can quote from Google's specification — a missing colon, a byte order mark, a field name that is not one of the four. Nothing else was counted as an error, which is why the totals below are smaller than a Lighthouse audit might suggest.

curl -s -o robots.txt -w "%{http_code} %{size_download}\n" \
  -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" \
  https://example.com/robots.txt

python3 -c "
raw = open('robots.txt','rb').read()
text = raw.decode('utf-8')
print('bytes', len(raw), 'bom', text.startswith(chr(0xfeff)))
known = ('user-agent','allow','disallow','sitemap')
for i, line in enumerate(text.splitlines(), 1):
    s = line.strip()
    if not s or s.startswith('#'):
        continue
    if ':' not in s:
        print('NO COLON    line', i, s[:60])
    elif s.split(':',1)[0].strip().lower() not in known:
        print('UNKNOWN     line', i, s[:60])
"

What "robots.txt is not valid" means in Google's own words

The failure mode is quieter than the message suggests. Google does not reject a file with one bad line; it discards that line and keeps reading. The robots.txt specification puts it in one sentence, and that sentence is the reason this article exists.

"Google ignores invalid lines in robots.txt files, including the Unicode Byte Order Mark (BOM) at the beginning of the robots.txt file, and use only valid lines." — Google Search Central, robots.txt specification, updated 2026-08-31 UTC, retrieved 2026-09-14

The same page defines what a valid line is and what a valid file is: "A valid robots.txt line consists of a field, a colon, and a value," and "The robots.txt file must be a UTF-8 encoded plain text file and the lines must be separated by CR, CR/LF, or LF." It also names the four fields Google supports, with an aside that matters more than the list: "other fields such as crawl-delay aren't supported."

What the 30 files actually contain

Ten of the 30 files hold more than 100 rule lines; Figma's single file is 100,975 bytes and 8,901 rule lines, which is 19 percent of the 500 KiB Google will read. Nothing in the panel came close to any limit. What the panel did contain is a long tail of lines Google drops without telling anyone.

Two entries in that table are worth naming. Wikipedia's file opens with a byte order mark, three bytes that Google's parser is documented to ignore; the mark turned its first line, a comment, into a field name the parser did not recognise, so the file's opening comment is simply not part of the file anymore. Stack Overflow's file is the one that answered a status code other than 200, and that is a different problem entirely — see below.

Field names Google does not read

Thirteen of the 30 files contain at least one field name outside Google's four. None of the three names we found causes an error message anywhere, because Google's behaviour with an unsupported field is to skip it — the line stays in the file, and the crawler never acts on it.

Content-Signal is the interesting case, because it is not a typo and not a leftover — it is a deliberate addition, and the sites using it know Google does not read it. We have a separate measurement of that field across a wider panel: content signals in robots.txt found 7 of 34 sites carrying it. The crawl-delay lines are the older story, and they behave differently from what most people assume: crawl-delay across 343 user-agent groups found that no crawl-delay line in that panel pointed at an AI crawler.

One file answers 418 to every client we tried

Stack Overflow's robots.txt returned HTTP 418 with 113 bytes of content, and it did so for our Chrome user agent, for our curl user agent, and for a Googlebot user agent. The body is a real robots.txt: a licence line, then User-agent: * and Disallow: /.

If that status code is what Google's crawler receives, the file's rules do not apply. Google's specification says of client errors: "Google's crawlers treat all 4xx errors, except 429, as if a valid robots.txt file didn't exist. This means that Google assumes that there are no crawl restrictions." A file that says "crawl nothing" and a status code that says "no restrictions" cannot both take effect, and the status code is what the crawler sees first.

We cannot close that loop from here, and we are not going to pretend otherwise. Our request came from one address; Google's crawler fetches from its own. A site that answers 418 to us may answer 200 to Google, and nothing in our data distinguishes those two cases. What we can say is narrower and still useful: a status code is part of your robots.txt configuration, and it is the part that overrides everything you wrote.

What this means for your own file

Run the two checks above against your own robots.txt and read the output, not the count of rules. A file with 400 lines and one unsupported field is in worse shape than a file with six lines that are all valid, because the 400-line file has one rule the crawler will never apply and nobody will email you about it.

1. Fetch your own robots.txt and confirm the status code is 200. Anything in the 400s means your rules are not being read at all.
2. Decode it as UTF-8 and print any line without a colon, and any line whose field name is not one of the four.
3. Check the first three bytes. If they are EF BB BF, you saved the file with a byte order mark and should re-save it without one.
4. Move anything you were relying on from an unsupported field into a supported one, or accept that the line is decoration.

• Test your robots.txt with the status code and the encoding, because those two decide whether the rest of the file exists at all.
• Do not assume a validator that shows no errors has read your file the way a crawler does; most validators check syntax, and the interesting failures are in the lines a parser drops silently.

One more thing belongs on the list, and it is the opposite of a fix. If you are using an unsupported field because you want a specific engine to honour it, keep it — Content-Signal is a reasonable example, since it is aimed at engines that are not Google. Just stop counting it as a control you have over Google Search. A crawler accessibility check will tell you what an AI crawler actually receives; your robots.txt is only one of the layers that decides the answer.

Common questions

How did you measure this?

One request per domain on 2026-09-14 with a desktop Chrome user agent, no JavaScript, redirects followed. We saved the response body as bytes and ran the two checks in the script above against the saved copy. The panel is the 30 sites used across this series, and three of them returned no usable homepage on the same day, which does not affect the robots.txt count because robots.txt was fetched from the domain root rather than from the homepage.

Does one invalid line break my whole robots.txt file?

No, and that is the problem. Google drops the line and keeps the rest, so a file with one unsupported field still works for every other rule in it. You get no error, no warning, and no report of any kind.

Is crawl-delay worth adding anyway?

Not for Google. Google's specification lists the four fields it supports and names crawl-delay as one it does not. If you want to slow Googlebot down, use the crawl rate setting in Search Console instead, which is a control Google actually reads.

Why did most of these large sites pass a strict check?

Because the checks that matter are easy to pass once you have done them once: a plain UTF-8 file with colons in it. The panel is 30 large sites with full-time platform teams, so it is the wrong sample for estimating how common these problems are on smaller sites. That is a limit of this survey, not a finding about the web.

What happens above the 500 KiB limit?

Everything after the limit is ignored, so the tail of a very long file simply does not exist for the crawler. Nothing in this panel was close: the largest file was 100,975 bytes, about a fifth of the limit. We did not test what a truncated file does in practice, because building one would mean publishing a robots.txt we know to be broken.