Vary header on 27 homepages: every one compresses, eight never declare it

FIELD TEST · 2026-09-07 · 27 homepages · one request each · who declares what changed the response

Sample and method: 30 homepages fetched once each on 2026-09-07 with a desktop Chrome user agent, redirects followed, no JavaScript executed, exit node in Japan. Three dropped out — stackoverflow.com, medium.com and www.reddit.com each returned 403 — leaving 27. Every response header was recorded as sent, including repeated fields.

All 27 homepages returned a compressed body. Only 19 named Accept-Encoding in a Vary header, so eight served a negotiated response without declaring the negotiation. Four of those eight sent no Vary header at all; the other four sent one that lists their framework's routing fields and nothing else. Not one homepage on this panel sent Vary: User-Agent.

How we measured this

One GET per site, desktop Chrome user agent rather than Googlebot, redirects followed, and the full response header set written to disk. Header names were lowercased for counting, values split on commas, and each field name kept as its own token. Repeated header lines were preserved rather than merged, because a server that sends two Vary lines is doing something different from a server that sends one line with two names in it.

# The same two lines, against your own homepage
curl -sI -H 'Accept-Encoding: br, gzip' https://example.com/ \
  | grep -i -E '^(vary|content-encoding):'

# Compressed body but no accept-encoding in Vary = the case counted below

Three limits, and the middle one is the one that matters most. One request per site and homepage only, so this is a snapshot of one address, not of a site. A single request also cannot show whether a cache in front of the site is behaving correctly — we recorded what was declared, and we did not test what any CDN actually stored or served to a second visitor. And we did not send varied request headers to see whether the response body changed, so a site that omits Accept-Encoding may still be serving correctly to everyone; what it is not doing is saying so.

Every homepage compressed, and eight never declared it

Compression is universal on this panel: 27 of 27 returned Content-Encoding, 16 with Brotli and 11 with gzip. That makes the second number easy to read, because the denominator is not in doubt. Every one of these responses was chosen partly on the basis of a request header, and 19 of 27 said so.

The eight split into two groups that look nothing alike. Four sent no Vary header whatsoever: about.gitlab.com, astro.build, stripe.com and www.wikipedia.org. The other four — nextjs.org, react.dev, supabase.com and vercel.com — did send a Vary header, listing exactly four fields, all of them belonging to the same framework's client-side routing: rsc, next-router-state-tree, next-router-prefetch and next-router-segment-prefetch. The response was compressed. Compression was not in the list.

RFC 9110 defines the field plainly: the Vary header "describes what parts of a request message, aside from the method and target URI, might have influenced the origin server's process for selecting the content of this response" (RFC 9110, section 12.5.5, read 2026-09-07). The specification's own example is the exact case these eight sites are in: a response carrying Vary: accept-encoding, accept-language "indicates that the origin server might have used the request's Accept-Encoding and Accept-Language header fields (or lack thereof) as determining factors while choosing the content for this response."

What the field buys is stated as a requirement on caches. A list of field names tells cache recipients that they "MUST NOT use this response to satisfy a later request unless the later request has the same values for the listed header fields as the original request". The same paragraph gives the one-line version: "Vary expands the cache key required to match a new request to the stored cache entry."

Vary is not a description of your server. It is an instruction to every cache between you and the reader.

The specification also says when a server ought to send it: an origin server "SHOULD generate a Vary header field on a cacheable response when it wishes that response to be selectively reused for subsequent requests." It allows the omission too, and names the tradeoff — Vary "might be elided when an origin server considers variance in content selection to be less significant than Vary's performance impact on caching". That is a real engineering choice. Whether four sites shipping a router-only Vary made it deliberately is not something one request can tell us.

Twenty-three Vary header values, eleven different shapes

Among the 23 sites that sent a Vary header there are 11 distinct sets of field names. The single most common shape is the plain one — nine sites send Accept-Encoding and nothing else. After that the panel scatters immediately.

Six sites carry the framework routing tokens: nextjs.org, react.dev, supabase.com, vercel.com, linear.app and www.figma.com. Two of those six also name Accept-Encoding, which is why the routing tokens are not by themselves the problem — linear.app ships eleven field names and Accept-Encoding is one of them.

Three sites put a CDN's or their own application's private field into the cache key. www.bbc.com sends X-BBC-Edge-Scheme, www.nytimes.com sends Fastly-SSL, and www.theverge.com sends x-user-state. None of these are request fields a search crawler will ever send, which is the point: they widen the cache key for the infrastructure that understands them and are inert everywhere else.

Four sites ship a header that repeats itself

Four homepages send a Vary value that names the same field more than once, or splits itself across two header lines. This is harmless — field names are matched case-insensitively and a duplicate adds nothing to the cache key — but it is a reliable sign that more than one layer is writing the same header.

railway.com is the interesting one. Its two lines arrive with different capitalisation, which is what an origin and an edge each writing their own header looks like from outside. github.com's single line is the longest on the panel and reads like an inventory of every front-end feature the site has shipped: X-PJAX, Turbo-Visit, Turbo-Frame, Accept-Language, Sec-Fetch-Site and six more.

Nobody varies on User-Agent, and one site varies on device anyway

Zero of 27 homepages sent Vary: User-Agent. That is the cleanest result in this measurement, and it lines up with what Google recommends rather than contradicting it. Dynamic serving — one address returning different HTML to phones and desktops — is the configuration that needs the field. Google describes it as a setup that "relies on user-agent sniffing and the Vary: user-agent HTTP response header to serve a different version of the HTML to different devices", and says plainly that "Google recommends Responsive Web Design because it's the easiest design pattern to implement and maintain" (Mobile-first Indexing Best Practices, Google Search Central, read 2026-09-07).

So an empty column here is the expected shape of a panel that builds responsively. One site is not in that column by a different route: www.notion.com sends X-Device-Class, accept-encoding. It varies by device class, using a field its own edge sets rather than the raw user agent string. We did not request that homepage as a phone, so we cannot say what would have come back.

What this means for you

The check is two lines long and the fix, where there is one, is a configuration line rather than a code change. Run it against your own homepage before assuming your stack handles this.

1. Request your homepage and print only Content-Encoding and Vary. If the first exists and the second does not name Accept-Encoding, you are in the group of eight.
2. Check whether anything in front of your origin is caching that response. A missing Accept-Encoding costs nothing when nothing is storing the response; it is the shared cache that can hand a compressed body to a client that did not ask for one.
3. Read the field list you do send, out loud. If it names your framework's routing fields and not the thing your CDN negotiates on, two layers are writing that header and only one of them knows about compression.
4. Leave User-Agent out unless you actually serve different HTML per device. On a responsive site it splits the cache for no benefit, and none of the 27 sites here send it.
5. Do not reach for Vary: *. The specification says a proxy "MUST NOT generate" it, and it tells every cache that no stored copy can be reused without asking you first.

The reason this belongs next to crawling rather than next to performance is that Vary decides which stored copy a shared cache hands to the next requester, and a crawler is one of those requesters. What compression a site negotiates in the first place is counted in content-encoding across 27 homepages, and the device-parity question behind the empty User-Agent column is worked through in mobile first indexing. To see what an AI crawler is handed when it asks for your page, use the AI crawler access check.

Common questions

How did you measure this?

One GET per homepage on 2026-09-07, desktop Chrome user agent, redirects followed, exit node in Japan, all response headers stored as received with repeated lines preserved. Values were split on commas and lowercased for counting. Three of the 30 sites returned 403 and were dropped before analysis, leaving 27. No JavaScript ran, and no second request with different headers was made.

Does the Vary header affect SEO?

Not as a ranking input, and no Google documentation describes it as one. It matters through delivery: it is the field that tells shared caches when a stored response may be reused, and Google names it in exactly one configuration, dynamic serving, where one address returns different HTML per device. On a responsive site the honest answer is that Accept-Encoding is the only name most homepages need.

What happens if Accept-Encoding is missing from Vary?

The documented risk is a shared cache reusing one stored copy for requests that did not negotiate the same way. We did not observe that happening to any of the eight sites — this measurement recorded declarations, not cache behaviour, and modern CDNs commonly handle compression as a special case regardless of what the header says. That is the reason eight sites can ship this and nothing visibly breaks.

Should I add Vary: User-Agent for mobile?

Only if you serve different HTML at the same address depending on the device. Google's mobile-first guidance ties that field to dynamic serving and recommends responsive design instead. Zero of the 27 homepages here send it, which is what you would expect from a panel that is responsive rather than device-switching.

Why do four sites list only their framework's routing fields?

We do not know. From one request we cannot tell whether the origin's header was overwritten downstream, whether the edge strips and rewrites it, or whether compression is handled at a layer that never consults the field. All four run the same framework and all four look identical from outside, which is a pattern rather than an explanation.