The robots.txt sitemap line: who declares one, and does it open
A robots.txt sitemap line is where a crawler looks without being told. Of 30 sites read on 2026-08-18, 26 declare one and four do not — and of the 26 first-declared files we fetched, one returned 403.
FIELD TEST · 2026-08-18 · 30 sites · single snapshot
Sample and method: the same 30 sites from our robots.txt, sitemap and homepage structure surveys. One robots.txt fetch each plus one fetch of the first sitemap it declares, browser user agent, 2026-08-18, from an exit in Japan. Same crawl as four other posts in this batch.
A robots.txt sitemap line is the one place a crawler looks for your sitemap without being told. Twenty-six of 30 sites have one; four do not. Of the 26 first-declared sitemaps we then fetched, 25 returned XML and one returned 403 — a declared address that the crawler being addressed cannot open.
How we measured it
Fixed before the crawl, unchanged after.
- Fetch
/robots.txton the final homepage host and record the status code. - Extract every line matching
Sitemap:, case-insensitive, and count them. - Fetch the first declared URL and record its status, size and content type.
Only the first declared sitemap per site, because the point was whether the declaration resolves, not to audit whole sitemap trees. That choice is also its limit, covered below.
How many sitemaps 30 sites declare
Most sites name exactly one. The counts at the top belong to newsrooms splitting archives by section and year.
| Declared | Sites | Which ones |
|---|---|---|
| None | 4 | github, stackoverflow, reddit, hacker news |
| Exactly one | 13 | nextjs, stripe, vercel, linear, cloudflare, wikipedia |
| Two to seven | 10 | figma 3, slack 3, railway 4, discord 6, wired 7 |
| More than ten | 3 | notion 12, nytimes 25, bbc 35 |
Naming none is a position, not an oversight, for at least two of the four. GitHub and Hacker News both run on URL patterns a crawler can walk from links alone. Whether that is the reason is not something we can confirm from outside.
Google supports the field, and only a handful of fields
The robots.txt specification page lists what Google reads: "Google supports the following fields (other fields such as crawl-delay aren't supported)", and names four — user-agent, allow, disallow, and "sitemap: the complete URL of a sitemap". Read 2026-08-18 at developers.google.com.
The word doing the work is "complete". A relative path is not a supported form, which is why a robots.txt sitemap entry takes a full URL including the scheme and host.
One declared sitemap answered 403
We fetched the first sitemap each of the 26 sites declares. Twenty-five returned 200 with XML. Wikipedia's declared address, en.wikipedia.org/w/rest.php/site/v1/sitemap/0, returned 403 to us.
A sitemap you declare and a sitemap a crawler can open are two different claims, and only one of them is checkable from your own dashboard.
We do not know whether Googlebot gets the same 403. That endpoint may be rate-limited or restricted by address range, and our request came from one browser user agent in one country. What we can say is that the check costs one command and nobody appears to run it.
The file itself is not always where you left it
Two things showed up that have nothing to do with sitemaps and everything to do with fetching robots.txt at all.
Stack Overflow answered our robots.txt request with 418, the joke status code reserved for teapots, which in practice is a bot-management system declining to talk. Twenty-nine of 30 returned 200.
Sizes and content types were also inconsistent among the sitemaps that did resolve. Medium's first file is 3,339,942 bytes served as binary/octet-stream; Canva's arrives as application/octet-stream; Discord's as text/plain. All parsed as XML anyway. None of this appears to break anything, and we did not test whether it slows anything down.
What this crawl cannot tell you
We opened one sitemap per site. A site can declare a valid first file and a broken fourth one, and this method would never see it. The three sites declaring more than ten files are exactly where that risk concentrates.
We also did not check whether any of these sitemaps were submitted in Search Console, which is the other discovery route and the only one that reports back. From outside a property there is no way to tell.
What to do about your own robots.txt sitemap line
Three checks, in order, none of which needs a tool.
- Open
yourdomain.com/robots.txtin a browser and confirm aSitemap:line exists with a complete URL on it. - Copy that URL into
curl -Iand confirm 200. Do it from a network that is not your office, so your own allowlist does not answer for you. - If the file is an index, open one child file too. The declaration only vouches for the first hop.
Declaring a sitemap is also how you deliver the freshness dates a crawler uses to prioritise, and how trustworthy those dates turn out to be is measured in the sitemap lastmod survey. Pushing a URL directly rather than waiting for a sitemap to be read is covered in the IndexNow setup guide. Checking that every declared file actually opens, on every site you own, is part of what QueryWin is being built to do.
Common questions
How did you measure this?
One robots.txt fetch per site, every Sitemap: line counted, then one fetch of the first declared URL. Browser user agent, 2026-08-18, one pass, no retries.
Do I need a robots.txt sitemap line if I already submitted the sitemap?
Submitting covers one search engine. The line in robots.txt is read by anything that fetches the file, which includes crawlers you never signed up with. Keeping both costs one line.
Can I list more than one sitemap?
Yes, and 13 sites here do. Three declare more than ten. Each line is independent, so a broken one does not invalidate the others — it just quietly delivers nothing.
Is a missing sitemap declaration a problem?
Not by itself. Four sites here declare none, and they are not obscure. It matters most for pages that are not reachable by following links, which is where a crawler has no other way in.
