Retry-After: how to ask a crawler to slow down, and what Google actually honors

The Retry-After header field tells a client how long to wait before it makes its next request. A server sends it with a 429, a 503, or a redirect to say "not now, try again in this many seconds, or after this date". For a crawler that is the difference between slowing down and backing off entirely, and it is the one rate-limit signal Google's own documentation names without ever promising to read.

What the retry-after header does, and what it does not

It is a scheduling hint attached to a response, not a command. RFC 9110 defines it in one sentence: "Servers send the 'Retry-After' header field to indicate how long the user agent ought to wait before making a follow-up request." That word "ought" is doing real work. A polite crawler respects it; nothing in the protocol forces one to.

Two response contexts matter here. With a 503 (Service Unavailable), Retry-After tells the client how long the service is expected to be down. With any 3xx redirect, it sets the minimum time before the client should follow the Location. The header is defined for both, and the syntax is the same in each case.

Retry-After is a request to wait, not a rule that enforces waiting.

A crawler is a client that fetches pages automatically on a schedule, such as Googlebot. For a site owner the practical use is narrow and specific: when your origin cannot serve a request — a deploy, an overloaded database, a maintenance window — you can return an error and use Retry-After to ask the crawler to come back later instead of hammering the same host. What you cannot do with it is cap the overall crawl rate. That is a different control, and this chapter is only about the per-response header.

It answers one bad moment. It does not set a permanent speed limit.

One boundary up front, because it shapes the rest of the chapter: Google's crawl documentation names 500, 503 and 429 as the codes that reduce crawl rate, but it does not state that Googlebot reads Retry-After. We checked that page on 5 October 2026 and the header is absent from it. We have not measured Googlebot's behavior against a live Retry-After, so this chapter teaches the syntax and the trade-offs, not a confirmed Google response.

Retry-After is often confused with Crawl-delay, the non-standard line some sites put in robots.txt. They solve different problems. Crawl-delay is a standing request for a fixed gap between requests, applied across a whole user agent, and Google has never supported it. Retry-After is a per-response answer to a single failed request, and it says "this particular moment is bad" rather than "always go slowly". If you need to protect the origin from an aggressive crawler every hour of the day, Retry-After is the wrong tool; if you need to survive a ten-minute deploy, it is the right one.

There is a second reason to keep the header narrow. Error responses should not be cached, and a Retry-After attached to a cacheable 200 invites a shared cache to store a delay that has already passed. Send it only on the responses that are genuinely transient, and make sure your CDN passes the header through rather than stripping it at the edge. A header that dies one layer above the origin is indistinguishable, from the crawler's side, from never having sent it.

Do it in this order

Four steps, each with a completion mark. The order matters because the status code carries most of the meaning and the header only refines it.

1. Pick the status code first. Decide whether the condition is temporary capacity (503), a rate limit you are imposing (429), or a redirect (3xx). Done when you can name the condition in one sentence.
2. Estimate the wait honestly. Use a number of seconds for a short, known window and an HTTP-date for a specific moment. Done when the value is one you would actually hold a client to, not a rounded guess.
3. Attach it to the response. Set the header on the same response that carries the status code — a header on a 200 is meaningless here. Done when a manual request shows both together.
4. Read your own log afterward. Confirm the client honored the wait rather than the header being cosmetic. Done when you can point at two requests spaced at least as far apart as the value you sent.

Step four is the test most teams skip. The header is cheap to set and easy to get wrong, and only the log tells you whether it changed anything.

The deliverable: status code, value format and a snippet

Three things to carry away. First, which status code to pair with the header. Second, the two legal value formats. Third, a minimal server snippet. Use the table to choose the code, then set the value.

The value has exactly two legal shapes, and mixing them is the most common mistake. A delay-seconds value is a non-negative integer: Retry-After: 120 means two minutes. An HTTP-date value is an absolute timestamp: Retry-After: Fri, 31 Dec 2026 23:59:59 GMT. RFC 9110 allows either; it does not allow a relative phrase, a unit suffix, or a bare decimal.

// Express: hold crawlers off for two minutes during a deploy
app.use((req, res, next) => {
  if (maintenanceWindow) {
    res.set('Retry-After', '120');
    return res.status(503).send('Temporarily unavailable');
  }
  next();
});

That snippet returns the header and the status together, which is the only correct pairing. A 503 without Retry-After still slows a compliant crawler; the header only tells it how long. A Retry-After without an appropriate status code does nothing at all.

What goes wrong

Three failures account for most of the bad Retry-After implementations we have seen described, and none of them shows up in a normal browser test.

1. The header is on the wrong response. Teams add Retry-After to every response as a blanket header, including 200s. Clients ignore it there, and the one response that needed it may not carry it.
2. The value is not one of the two legal formats. A value such as 2 minutes or 1.5 is not parseable. Send 120, or send a full HTTP-date.
3. The wait is a lie. A 503 that clears in five seconds with a Retry-After: 3600 teaches a compliant client to stay away for an hour. The code and the wait should describe the same event.

Ask for the wait you actually want, because a client that trusts you will take you at your word.

Common questions

Does Googlebot respect Retry-After?

Google does not say so in its crawl documentation, and we have not tested it, so we cannot promise it does. What Google does document is that a significant number of 500, 503 or 429 responses lowers the crawl rate for the whole hostname. Treat Retry-After as a refinement of that signal, not as a control Google publishes a contract for.

Should I use 429 or 503 to throttle a crawler?

Use 503 when the origin genuinely cannot serve requests, and 429 when you are deliberately limiting request rate. Google treats both as server errors for crawl-rate purposes, so the choice is about telling a human operator the truth. Do not use 403 or 404 for rate limiting; Google's status-code page states plainly that 4xx codes except 429 have no effect on crawl rate.

How long should the wait be?

As short as the situation truly allows, up to about a day for a planned outage. Google warns that serving errors for longer than one to two days can get a URL dropped from the index, so a long Retry-After is not a safe substitute for restoring the page. If the disruption will last longer, fix the origin instead.

How do I check that a crawler obeyed it?

Look for two requests from the same user agent spaced at least as far apart as the value you sent. That is the method in reading your server log to see who actually came; the header's effect is invisible from a browser.

Will Retry-After get my page dropped from the index?

A short one will not. Google's warning is about serving errors for longer than one to two days, not about the header itself: "if Googlebot observes these status codes on the same URL for multiple days, the URL may be dropped from Google's index." The header only tells the crawler when to return; the status code is what Google counts. Keep the window short and the value honest and the two stay aligned.

Does it work through a CDN?

Only if the CDN forwards the header. Most edge networks pass unknown response headers through by default, but caching rules and header-rewrite rules can remove it. Test with a direct request to the origin and a request through the edge, and compare the two. If the edge adds its own retry logic, which several providers do for overload, your origin's value may never reach the crawler at all.

What about Bing and other crawlers?

The header is part of HTTP, not of any one search engine, so every compliant client can read it. Whether a given crawler acts on it is that operator's choice, and we have not read a public commitment from any of them. The safe assumption is that a well-behaved crawler will at least not hammer a host that keeps returning 503; the specific wait is a hint you should not build a load plan around.

The boundary worth repeating: this header changes how a polite client schedules its next request and nothing more. It will not repair a slow origin, and it will not override a crawler that chooses to ignore it. If the underlying question is whether crawling volume is even a problem for your site, start with whether crawl budget is your problem; and if you want the crawl you do serve to turn into pages that get indexed and queried, that is the part QueryWin works on.