What AI crawlers get from JavaScript sites: 342 to 25,082 characters
What AI crawlers get from JavaScript sites varies 73-fold. We fetched 15 well-known homepages as an AI crawler: 14 returned 200, and the readable text ranged from 342 to 25,082 characters.
FIELD TEST · 2026-08-15 · 15 sites · single snapshot
Sample and method: 15 well-known public sites, one homepage request each, sent with the OAI-SearchBot user agent from a single machine on 15 August 2026. Scripts and styles stripped, remaining text measured in characters.
What AI crawlers get from JavaScript sites varies by a factor of 73. Fourteen of the fifteen returned 200, and the readable text in those responses ranged from 342 characters to 25,082 — across sites that all look complete in a browser. One site, airbnb.com, returned 403 and served nothing at all. The status code told us almost nothing; the character count told us everything.
How we measured it
One request per homepage, no rendering, no retries. We removed <script> and <style> blocks, stripped the remaining tags, collapsed whitespace, and counted characters. We also recorded whether an h1 existed in the delivered HTML and whether any JSON-LD block was present.
This is deliberately the crudest possible measurement. It is exactly what a crawler that does not execute JavaScript receives, which is the point.
What AI crawlers get from JavaScript sites, site by site
Sorted by readable text, largest first. The raw byte column is there to show how little the two have to do with each other.
| Site | Raw bytes | Readable text | h1 |
|---|---|---|---|
| webflow.com | 624,176 | 25,082 | 2 |
| framer.com | 2,150,677 | 12,708 | 1 |
| linear.app | 1,263,291 | 9,526 | 1 |
| supabase.com | 1,326,876 | 8,555 | 1 |
| slack.com | 250,889 | 8,286 | 1 |
| stripe.com | 658,126 | 7,342 | 2 |
| railway.com | 461,551 | 7,301 | 1 |
| nextjs.org | 315,975 | 7,110 | 1 |
| developer.mozilla.org | 121,316 | 6,723 | 1 |
| discord.com | 169,856 | 4,550 | 1 |
| figma.com | 1,675,088 | 3,932 | 1 |
| vercel.com | 3,012 | 2,980 | 0 |
| notion.com | 218,688 | 2,594 | 1 |
| canva.com | 8,008 | 342 | 1 |
| airbnb.com | 403 — refused | ||
Bytes are not content
The ranking by raw size and the ranking by readable text are almost unrelated. framer.com sent 2,150,677 bytes and yielded 12,708 characters of text — about six tenths of one percent. developer.mozilla.org sent 121,316 bytes, seventeen times less, and yielded 6,723 characters, only half as much text.
A crawler paying to download two megabytes to find twelve thousand characters is being asked to work hard for very little. Nothing in the response signals that.
The two outliers are the interesting part
canva.com returned 8,008 bytes containing 342 readable characters. That is a page which, to a client that does not run scripts, is essentially blank. Anyone browsing it sees a full marketing homepage.
vercel.com is odder and we are not going to pretend we know what happened. It returned 3,012 bytes, of which 2,980 were text — a 98% text ratio, the highest in the set — and no h1 at all. That combination usually means we were served something other than the normal homepage for this user agent. We did not chase it down, and a single request is not enough to claim anything about it.
One site refused outright
airbnb.com returned 403. That is one refusal in fifteen requests, and it is worth stating plainly because a separate probe we ran the same day against eleven other well-known sites returned 200 every time. Blocks at the edge are real but, in this sample, uncommon.
We cannot tell you why. A 403 does not explain itself, and we did not test from other addresses, so we cannot separate a user-agent rule from an address-based one.
What this does not show
This is one request per site, from one machine, on one day, to homepages only. It does not measure whether any engine actually cited these sites, whether the deep pages behave like the homepages, or whether a real crawler arriving from its own address gets the same answer. Homepages are also the least representative page on most sites — they are usually the most hand-built.
It measures exactly one thing: what arrived, once. Everything else here is inference from that.
Common questions
How did you measure this?
One curl per site with the OAI-SearchBot user agent, scripts and styles removed, tags stripped, characters counted. The method is written up step by step in can AI crawl JavaScript, so you can run it against your own site and compare.
Does a low character count mean the site has an SEO problem?
Not necessarily, and not for Google specifically, which renders JavaScript on a delayed second pass. It means the content is not present for clients that do not render — which is most of the crawlers feeding AI answers.
Why homepages and not article pages?
Because homepages are comparable across companies that otherwise share nothing. It is also the main limitation of the sample, and a deep-page version is the obvious next measurement.
The one number to take away
Fourteen sites, all returning 200, all looking complete in a browser, delivering between 342 and 25,082 characters of readable text. If you have never checked which end of that range your own site sits at, the status code has been telling you nothing. Making the fix, publishing it and pushing it for indexing is the part that changes the answer.
