What AI crawlers actually see on your site: 85 requests, all 200, 14× apart
What AI crawlers actually see on your site is not what your browser shows you. We fetched five live sites with 16 crawler user agents: all 85 requests returned 200, and the readable text behind them differed by 14×.
FIELD TEST · 2026-08-14 · 5 sites · 85 requests
Sample and method: 5 live sites, 16 AI crawler user agents plus one browser baseline, one homepage request each on 2026-08-14, plus
robots.txtandllms.txtfor every site, plus 10 interior URLs sampled from each sitemap. Single snapshot, not monitoring.
What AI crawlers actually see on your site is not what your browser shows you, and the status code will not tell you the difference. We pointed 16 crawler user agents at five sites we run. Every one of the 85 requests came back 200. Then we measured what was inside those 200s: the amount of readable text ranged from 961 characters to 13,393, a 14× spread across sites that all looked equally healthy. One site returned no <h1> at all. One returned its homepage HTML when a crawler asked for /llms.txt.
A free reachability check would have given all five a green light.
Start with the one that lies
A crawler asks hailuoshe.com for /llms.txt:
GET https://www.hailuoshe.com/llms.txt (as GPTBot)
→ 200 OK
→ body starts: <!doctype html><html lang="en"> ...
→ 1,380 bytes of HTMLThere is no llms.txt at that path. A single-page app catch-all route answered for a file that does not exist, and answered convincingly: 200, non-empty, plausible length.
In the same run, cuotiguanjia.com returns an honest 404 for the identical path.
The 404 is the better outcome. It tells a crawler nothing is there and ends the fetch. A 200 full of markup gets accepted as a valid file, wastes crawl budget, and can be parsed as a garbled site description.
Every "check whether AI can crawl your site" tool shows this as green, because the status code is genuinely fine.
How we tested
Each user agent string was copied from the operator's own documentation, not guessed:
| Operator | User agents used |
|---|---|
| OpenAI | GPTBot, OAI-SearchBot, ChatGPT-User |
| Anthropic | ClaudeBot, Claude-SearchBot, Claude-User |
| Perplexity | PerplexityBot, Perplexity-User |
| Google / Microsoft / Apple | Googlebot, Bingbot, Applebot |
| Others | Bytespider, CCBot, meta-externalagent, Amazonbot, DuckAssistBot |
One request per user agent per homepage, no cookies, no retries:
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
-A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" \
https://example.com/Then, for the body measurement, we stripped <script>, <style> and comments, removed the remaining tags, and counted what was left. That number is roughly what an answer engine has to work with.
Our first pass recorded only status codes and response sizes. That is what a checker does, and it told us everything was fine. The byte counts are the only reason we kept going: 18 KB against 205 KB is not a design choice.
What AI crawlers actually saw on these five sites
| Site | Readable text | Text as % of HTML | <h1> | JSON-LD blocks | AI crawlers named in robots.txt | llms.txt |
|---|---|---|---|---|---|---|
| sizemarker.com | 13,393 chars | 7.3% | 1 | 5 | 13 | 20.7 KB |
| biaojixia.com | 4,744 chars | 3.2% | 1 | 5 | 14 | 14.3 KB |
| byerisk.com | 3,665 chars | 1.7% | 1 | 1 | 14 | 3.3 KB |
| cuotiguanjia.com | 2,631 chars | 6.1% | 1 | 0 | 14 | 404 |
| hailuoshe.com | 961 chars | 5.2% | 0 | 0 | 14 | HTML |
Three things in that table are worth more than the status codes.
The text ratio. byerisk.com ships 205 KB of HTML to deliver 3,665 characters of prose. That is 1.7% signal. The rest is markup, inlined state and framework payload that an answer engine has to walk past before it finds a sentence worth quoting. Nothing about that is broken, and every reachability tool on the internet will call it a pass.
The missing heading. hailuoshe.com returns no <h1>. A crawler landing there gets 961 characters with no structural anchor and no indication of what the page is about beyond the <title>.
Structured data splits the group. Three of the five carry JSON-LD, two carry none. Among the three, FAQPage is the most common type, and it happens to be the block an answer engine can lift whole.
The counterintuitive part: the homepages were the problem, not the interior pages
We expected the opposite. Interior pages are numerous, templated, easy to neglect. So we sampled 10 URLs from each site's sitemap, 50 in total, and ran the same user agent against them.
It went the other way. Apart from the 961 characters on the hailuoshe.com homepage, all 49 remaining URLs returned more than 1,800 characters of text, topping out at 9,481. The interior pages were the best-rendered part of every site.
Which makes sense on reflection. Interior pages are built around content, so the content has to exist in the HTML. Homepages are built around interaction and visual design, and their words are often assembled by the front end. Testing only the homepage and testing only the interior pages both give you the wrong answer, and most people test only the homepage.
What this means for you
Run the check in this order. Each step only matters if the one above it passed.
- Status code per crawler user agent — the cheap part, and where most tools stop
- Readable text in the response body, with scripts stripped — if this is near zero, the status code is decoration
- One
<h1>present, and it describes the page robots.txtnames AI crawlers explicitly instead of leaving them to the wildcard/llms.txteither exists as text or returns a real 404
Two that go the other way. Get these wrong and the five above tell you nothing:
- Don't accept a 200 on any path your framework can catch-all — check the content type and the first bytes
- Don't test only the homepage, and don't test only the interior pages — they fail for different reasons
The one-line version: a crawler that can reach you and a crawler that can read you are two different tests, and only the second one decides whether you get quoted.
What we did not measure
This test spoofs user agents from an ordinary IP address. Real crawlers arrive from published IP ranges, so a site that filters by IP would still block them while passing everything here. We measured whether a server refuses a user agent string, which is the same thing the free checkers measure, and no more.
It also says nothing about whether these crawlers actually come. Reachable is not the same as visited, and the only place that answer lives is a server log. 30 days of server logs across the same sites are being pulled now.
One result we cannot explain yet: on two sites, Bingbot and Applebot received a response 1,581 bytes smaller than every other user agent, consistently. Same status, same path. We have not worked out why, and would rather say so than guess.
Frequently asked questions
How did you test this? 16 crawler user agents copied from operator documentation, one homepage request each across 5 sites, plus 10 interior URLs per site from the sitemaps, on 2026-08-14. Body text measured after removing scripts, styles and comments. Raw numbers are in the table above; nothing is modelled or extrapolated.
Does a 200 mean AI can read my site? No. It means your server did not refuse that user agent. Whether there is anything readable in the response is a separate measurement, and across these five sites the answer ranged from 961 to 13,393 characters.
My site is client-side rendered. Is it invisible to AI? Not automatically, but you should verify rather than assume. Fetch a page with curl and a crawler user agent, strip the tags, and read what is left. If that is a few dozen characters, that is what the crawler gets.
Do I need an llms.txt? Less urgent than making sure the path does not return HTML. A 404 is a clean answer. A 200 that serves your homepage is worse than having no file.
How many AI crawlers should robots.txt name? Naming them is about intent, not access. The wildcard rule already covers them. Naming them means you have decided per operator instead of inheriting a default, and it makes the file readable to whoever inherits it next.
We build QueryWin to find the searches you should be winning on Google and in AI answers, work out why you are not, and ship the fix to your site. If you have not run the first check yet, the handbook covers how to check if AI can read your site step by step.
