Last modified header: who still sends one, and does it save a fetch

FIELD TEST · 2026-08-18 · 30 sites · single snapshot

Sample and method: the same 30 sites from our robots.txt, sitemap and homepage structure surveys. Homepage response headers plus two conditional requests each, browser user agent, 2026-08-18, from an exit in Japan. Three sites refused us, so counts are out of 27. Same crawl as four other posts in this batch.

A crawler that already has your page can ask whether it changed instead of downloading it again. That only works if you gave it something to ask with. Of 27 homepages, 13 send neither a last modified header nor an ETag, so no conditional request is possible at all. Where an ETag existed it worked 9 times out of 10. Where only a date existed it worked 3 times out of 7.

How we measured it

Fixed before the crawl, with one correction we made mid-run and kept in the record.

1. Fetch each homepage once and record Last-Modified, ETag and Cache-Control.
2. Re-request with If-None-Match set to the ETag the site had just sent.
3. Re-request with If-Modified-Since set to the Last-Modified value the site had just sent.

Step three is the correction. Our first pass sent one fixed timestamp to every site, which produced a headline number of 1 site out of 27 — meaningless, because a site whose content really did change after that timestamp is right to answer 200. Echoing each site's own value back is the only version of this test that means anything.

What 27 homepages send

Half the sample sends nothing a crawler can validate against.

The pattern is not about company size. Cloudflare, Stripe and Notion send nothing; Wikipedia and MDN send both. What separates them is closer to how the page is built and cached than to how much engineering is behind it.

Which conditional requests actually returned 304

Sending the header is a promise. Honouring it when the crawler comes back is the part that saves anyone anything.

The three that answered 304 to a date were mozilla, webflow and wikipedia. Each returned an empty body, which is the whole point: the crawler learns nothing changed and pays almost nothing to learn it.

Among the four that answered 200, two are easy to defend. The New York Times had stamped its homepage minutes earlier and genuinely changes all day. Framer's timestamp was also fresh. Discord is the awkward one — its stamp said 14 August, four days before our request, and it still sent 170,022 bytes.

We cannot tell whether any of these servers ignored the header. From outside, "the page changed in the seconds between our two requests" and "this server does not implement conditional requests" produce the identical response. That distinction needs server logs we do not have.

Google says use ETag, and our numbers agree

Google's crawling documentation states its preference plainly: "We strongly recommend using ETag because it's less prone to errors and mistakes (the value is not structured unlike the Last-Modified value). And, if you have the option, set them both." Read on 2026-08-18 at the Crawling December caching post on developers.google.com, published 2024-12-09.

The same post gives the number that puts our 27 sites in context. Google writes that the share of its fetches that could be served from cache "has decreased: 10 years ago about 0.026% of the total fetches were cacheable, which is already not that impressive; today that number is 0.017%."

Two hundredths of one percent. That is how often the web answers "nothing changed" to a crawler that asked politely.

Our small sample lands on the same side of the argument for a different reason. The date-based test failed for four sites out of seven; the tag-based test failed for one out of ten. A last modified header is a claim the server has to keep consistent across every caching layer in front of it. An opaque token is not.

What this crawl cannot tell you

One request pair per site, on one day, from one country. We did not repeat the test hours later, so a server that answers 304 in a quiet moment and 200 under load looks identical to one that always answers 304.

We did not test whether Googlebot itself gets the same treatment we did. Some edge configurations vary by user agent, and we sent a browser one throughout. Everything above describes what a client gets, not what a named crawler gets.

What to do about your own last modified header

Three things worth checking, in this order.

1. Run curl -I on your homepage and look for ETag and Last-Modified. Neither present is the 13-out-of-27 case, and it is the cheapest one to fix.
2. Take whichever value you got, send it straight back as If-None-Match or If-Modified-Since, and confirm you get 304 with an empty body.
3. If your CDN strips or rewrites these headers, fix it there. The origin being correct is worth nothing if the edge answers first.

This matters most for sites with many pages that rarely change — documentation, archives, product catalogues. On a five-page site the saving is theoretical. Freshness dates you publish in a sitemap are a separate signal with separate problems, measured in the sitemap lastmod survey. What a crawler does with a page it decides to fetch anyway is covered in the soft 404 survey. Watching these headers across a whole site instead of one homepage is part of what QueryWin is being built to do.

Common questions

How did you measure this?

One homepage fetch per site to read the headers, then one conditional request per header type, echoing the site's own value. Browser user agent, redirects followed, 2026-08-18. Status code and body length recorded for each.

Do I need both ETag and Last-Modified?

Google recommends setting both and says it uses the ETag when both are present. Our sample suggests the same ranking for a plainer reason: the tag-based test worked more often.

Does a 304 help my rankings?

Not something we tested, and not a claim Google makes. The documented benefit is on crawling and bandwidth. Treat it as an efficiency fix, not a ranking one.

Why did only 3 of 7 last modified header tests work?

Two of the four failures had genuinely fresh timestamps and were probably right to send the page. The other two are unexplained from outside, and we did not guess.