Content-Encoding on 27 homepages: 16 brotli, 11 gzip, zero zstd

FIELD TEST · 2026-09-02 · 27 homepages · three requests each · Accept-Encoding negotiation

Sample and method: 30 homepages fetched on 2026-09-02 with a desktop Chrome user agent, redirects followed, no JavaScript executed. Three dropped out — stackoverflow.com and medium.com returned 403, www.reddit.com returned an 8,393-byte shell — leaving 27. Each of the 27 was then requested three times with a different Accept-Encoding, and the bytes on the wire were counted without decoding.

Every one of the 27 homepages compressed its HTML when asked. Content-Encoding came back as br on 16 sites and gzip on 11; zstd came back zero times, even though every request advertised it. Ask for nothing and all 27 hand over the raw file: 18,593,772 bytes against 2,490,063 for the same 27 pages fully negotiated, a factor of 7.5.

How we measured it

Three GETs per site on 2026-09-02, same user agent and same redirect handling each time, differing only in one request header:

Bodies were read with decoding switched off, so every byte count below is what crossed the network, not what the parser saw. That is the only way to compare the three runs on the same scale.

# The same three reads on one of your own pages
for ae in 'gzip, deflate, br, zstd' 'gzip' 'identity'; do
  curl -s -o /dev/null -H "Accept-Encoding: $ae" \
    -w "$ae -> %{size_download} bytes\n" https://example.com/
done

Two limits worth stating. The three requests were seconds apart, not simultaneous, and several of these pages change between requests — on the 11 gzip sites the full and gzip runs used the same algorithm and still differed by up to 1,316 bytes (www.nytimes.com), which is page variance, not compression. So treat anything under about two percent as noise. And we measured transfer only: no render, no paint, no timing.

content-encoding is br on 16 sites and gzip on 11

Nobody in the batch declined. That is the first result and it is a boring one, which is the point — HTML compression is settled, and any site still sending uncompressed HTML in 2026 is an outlier this sample did not catch.

The split matters more than the total. Brotli is not the default everywhere: 11 of 27 sites, including github.com, stripe.com, www.wikipedia.org, www.bbc.com and www.nytimes.com, returned gzip to a request that explicitly offered brotli. Those are not small engineering teams. Whatever the reason, offering br does not guarantee getting it.

zstd is the other half of that sentence. Every request in the full run advertised zstd, and not one of the 27 responded with it. MDN documents the value as a Content-Encoding token defined in RFC 8878 (Content-Encoding, read 2026-09-02). On these 27 homepages it is documented and unused.

Brotli buys 31.6% over gzip on the sites that use it

Comparing the full run against the gzip run on the 16 brotli sites: 2,071,362 bytes become 1,417,422. That is 653,940 bytes saved across 16 homepages, or 31.6% off the gzip figure. The spread inside that average is very wide.

Sixty-four percent at one end and four percent at the other, on the same swap between the same two algorithms. Compression level is a server setting and brotli has eleven of them, so a low-effort brotli can land near gzip. We did not measure which level any of these sites uses and cannot separate that from differences in the HTML itself.

Ask for no compression and you download 7.5 times as much

All 27 sites honoured Accept-Encoding: identity and sent the raw file. None of them ignored the request and compressed anyway. Summed across the panel that is 18,593,772 bytes uncompressed against 2,490,063 fully negotiated.

Per site the multiplier runs from 3.6x on stripe.com to 15.4x on www.theverge.com, with a median of 7.1x. Four sites cross ten: www.theverge.com at 15.4x, supabase.com at 14.2x, www.cloudflare.com at 12.5x and www.framer.com at 11.6x.

This is the number that matters for anyone writing their own checker. Google states plainly that "Google's crawlers and fetchers support the following content encodings (compressions): gzip, deflate, and Brotli (br)" and that support "is advertised in the Accept-Encoding header of each request they make" (Google Crawler Overview, read 2026-09-02). A homemade script that forgets that header does not behave like Googlebot. It behaves like the identity column.

Compression is negotiated, not configured. The client that forgets to ask is the one that pays, and on this panel it pays seven times over.

What this means for your site

Two things to check on the server, one on anything you crawl with.

1. Confirm your origin actually returns br to a request that offers it. Eleven sites here offered nothing better than gzip to a browser-shaped request, and every one of them looks correctly configured from the outside.
2. If you already serve brotli, look at the compression level before adding anything else. The gap between 4% and 64% in the table above is a settings gap, not an algorithm gap.
3. Send Accept-Encoding from every script that touches your site. Same page counts, seven times the bandwidth, and your logs will show a client that looks nothing like a real one.

Transfer size is one of the three numbers people mix together when a site feels slow. The raw weight of the HTML is measured in average web page size across the same panel, and the wait before the first byte in time to first byte. Whether any of it is worth your attention depends on a question answered separately in is crawl budget your problem. To see what a fetch of your own pages returns before any of this is tuned, see how QueryWin reads a page.

Common questions

How did you measure this?

Three GETs per homepage on 2026-09-02 with the same desktop Chrome user agent, differing only in the Accept-Encoding header: gzip, deflate, br, zstd, then gzip, then identity. Response bodies were read with automatic decoding disabled and the byte counts recorded. Redirects followed, no JavaScript executed, exit node in Osaka, Japan.

Is brotli always smaller than gzip?

On these 16 sites it was smaller every time, by between 4.2% and 64.4%. It is not guaranteed in general: brotli at a low compression level on already-minified HTML can land within noise of gzip, and we did not measure any site's level.

Why did no site return zstd?

We do not know. Every request advertised it and 27 of 27 chose something else, which is consistent with origins and CDNs not having it enabled, but this test cannot distinguish "not supported" from "supported and not preferred".

Does content-encoding affect SEO?

Not directly, and nothing here measures rankings. What it changes is how many bytes a crawler spends on the same page, which is a bandwidth question rather than a quality one. Google's own crawlers advertise gzip, deflate and br and take what you offer.

Should I worry that 11 sites are still on gzip?

Not on their behalf. The point of the 11 is that a site can offer good engineering everywhere else and still leave this one negotiation on its default — which is why reading the header on your own origin beats assuming the CDN handled it.