Crawl-delay: four lines in 343 user-agent groups, and none of them names an AI crawler

FIELD TEST · 2026-08-19 · 30 sites · single snapshot

Sample and method: the same 30 sites used in our robots.txt, sitemap, homepage structure and request chain surveys. One robots.txt request each, browser user agent, 2026-08-19, from an exit in Japan. One site refused us, so the parsed set is 29 files.

Across 29 robots.txt files containing 343 user-agent groups, we found four crawl-delay lines. Four. Three of them name a single specific crawler, one applies to everything, and not one of the four names an AI crawler — even though 13 of these 29 files name at least one AI crawler somewhere else in the file.

How we measured it

Fixed before the crawl, unchanged afterwards.

1. Request https://<host>/robots.txt once with a desktop browser user agent.
2. Parse it into groups the way the standard describes: consecutive User-agent lines open one group, and the first non-agent line closes the header.
3. Record every group that carries a Crawl-delay value, along with the agents that group applies to.

Stackoverflow.com answered its robots.txt request with 418, so it is excluded from all counts here. That is the same response it gave us on the previous run, which at least makes it consistent.

Four lines in 343 groups

Here they are, in full. This is not a sample of the crawl-delay lines we found — it is all of them.

The pattern is legible. Hacker News, which is a single small server behind a very large audience, throttles everyone at 30 seconds. The other three throttle one named commercial crawler each, and leave everything else alone. Nobody in this sample uses crawl-delay as general policy.

Google does not support the directive at all

This is stated in one line in Google's robots.txt documentation, inside the list of what it does read: "Google supports the following fields (other fields such as crawl-delay aren't supported)". The four fields it does support are user-agent, allow, disallow and sitemap. This line and the crawl rate guidance quoted below were read on 2026-08-19, on the robots.txt and reduce-crawl-rate pages at developers.google.com.

So the 30-second line on Hacker News does nothing to Googlebot. It may do something for other crawlers that implement it — we did not test which ones honour it, and a robots.txt file cannot tell you that. It is one of the few things in this area you can only learn from your own server logs.

A directive that the largest crawler ignores is not a rate limit. It is a request, addressed to whoever happens to read it.

What Google says to do instead

There is a documented alternative, and it is more aggressive than most people expect: "If you need to urgently reduce the crawl rate for short period of time (for example, a couple of hours, or 1-2 days), then return 500, 503, or 429 HTTP response status code instead of 200 to the crawl requests."

It comes with a hard time limit and a stated consequence. Google warns: "We don't recommend that you do this for a long period of time (meaning, longer than 1-2 days) as it may have a negative effect on how your site appears in Google products. For example, in case of Search, if Googlebot observes these status codes on the same URL for multiple days, the URL may be dropped from Google's index."

Recovery is automatic — "Once the number of these errors is reduced, the crawl rate will automatically start increasing again" — and there is a slower escape hatch for sites that cannot serve errors: a form, with the caveat that "it may take several days for the request to be evaluated and fulfilled." You also cannot use it to ask for more crawling.

Naming a crawler and throttling it are different habits

The files that name the most crawlers are not the files that throttle them. The Verge's robots.txt carries 106 user-agent groups and zero crawl-delay lines. The New York Times has 57, the BBC 40, Wikipedia 34, Figma 21.

These are the same publishers we found earlier blocking AI crawlers outright. Read the two findings together and the operating model is clear: when these sites want a crawler slowed down, they do not slow it down. They block it. Thirteen of the 29 files name at least one AI crawler, and every one of those names appears in an allow or disallow rule, never in a crawl-delay.

What to do about crawl-delay on your own site

Two habits worth keeping, two worth dropping.

• Keep an existing crawl-delay line if a specific non-Google crawler is genuinely hurting you. It costs nothing and some crawlers do read it.
• Solve real load problems at the server, with caching or a rate limit that returns a status code, not with a text file that four crawlers might honour.
• Do not add crawl-delay expecting Googlebot to slow down. The documentation says it is not supported, in those words.
• Do not serve 429 or 503 for more than a day or two. Google says the URL may be dropped from the index if it sees them for multiple days.

If your actual worry is that AI crawlers are costing you bandwidth, the lever is not delay, it is access — which crawler is allowed in at all, and what it gets when it arrives. Checking what one of them actually receives from your pages is what the AI crawler accessibility checker is for.

How to write the access rules themselves, bot by bot, is in robots.txt for AI crawlers. What else these same 30 files declare, and how many point at a sitemap, is in the robots.txt sitemap survey.

Common questions

How did you measure this?

One robots.txt request per site on 2026-08-19, browser user agent, exit in Japan. We parsed each file into user-agent groups and recorded every group carrying a Crawl-delay value. Twenty-nine of 30 files parsed; stackoverflow.com returned 418.

Does Google respect crawl-delay?

No. Google's robots.txt documentation lists the fields it supports and states that other fields, crawl-delay among them, are not supported.

Do AI crawlers respect crawl-delay?

We do not know, and this crawl cannot answer it. Finding out means watching your own server logs for a specific crawler before and after adding the line.

How do I actually slow Googlebot down?

Return 500, 503 or 429 to crawl requests for a short period, per Google's own guidance, and stop within a day or two. There is also a request form, which takes several days and cannot be used to ask for more crawling.

Is a crawl-delay of 30 seconds too high?

For a crawler that honours it, 30 seconds is roughly 2,880 pages a day, which is fine for a small site and impossible for a large one. Hacker News can afford it because its content lives on one page.