Serving markdown to AI crawlers: 3 of 6 docs sites already do
Anthropic, Next.js and Cloudflare return text/markdown when you append .md to a docs URL. One site returns HTML from its .md path while reporting 200 — check the content type, not the status code.
RULE CHANGE · 2026-08-15 · 6 documentation sites tested
Sample and method: we requested a Markdown variant of six documentation sites two ways — by appending .md to the URL, and by sending Accept: text/markdown — with the OAI-SearchBot user agent on 15 August 2026, and read the returned content type.
Serving markdown to AI crawlers is already shipping quietly: Anthropic, Next.js and Cloudflare all return text/markdown when you append .md to a docs URL. Cloudflare goes further and honours Accept: text/markdown on the normal URL. Two sites returned 404, and one returned HTML from its .md path — which is the failure mode worth knowing about.
The two mechanisms
They are different in a way that matters. One asks the crawler to know a convention; the other requires nothing of it but a header it may already send.
| Mechanism | How it works | Catch |
|---|---|---|
| URL suffix | /docs/page also answers at /docs/page.md | The client has to guess the convention exists |
| Content negotiation | Same URL, Accept: text/markdown returns Markdown | Needs server-side support and correct caching by variant |
Serving markdown to AI crawlers: who already does
Requesting the .md variant of each documentation entry point. The content type column is the one that matters.
| Site | .md suffix | Content type returned |
|---|---|---|
| docs.anthropic.com | 200 | text/markdown |
| nextjs.org | 200 | text/markdown |
| developers.cloudflare.com | 200 | text/markdown |
| docs.railway.com | 404 | — |
| supabase.com | 404 | — |
| linear.app | 200 | text/html |
On the Accept header test, Cloudflare's docs returned text/markdown from the ordinary URL. Next.js returned text/html — it supports the suffix but not the negotiation.
The 200 that means nothing
linear.app/docs.md returned 200 and text/html. That is not Markdown support; that is the application's catch-all route answering an unmatched path with the app shell.
This is the same trap that catches people publishing llms.txt on a single-page app, and it is worth stating as a general rule: on a client-rendered site, a 200 proves only that some route matched, never that the file you meant exists. Check the content type, not the status code.
Why serve Markdown at all
Because the conversion is lossy and you are currently making someone else do it. An HTML page wraps its content in navigation, scripts and layout; anything consuming it has to strip all that back out and guess at the structure. Serving Markdown hands over the version that was already clean before it got wrapped.
The honest caveat: we do not know that any AI crawler requests these variants. Three well-resourced documentation teams have decided it is worth doing, which is evidence about their judgement, not about crawler behaviour. Confirming it needs server logs showing requests for .md paths, and we do not have logs for anyone else's site.
If you want to do it
Start with the suffix, because it is static-file simple. Most documentation generators already have the Markdown source on disk; the work is routing a second URL to it with the right content type.
# Verify a site's support, both ways
curl -sI https://example.com/docs/page.md | grep -i content-type
curl -sI -H "Accept: text/markdown" https://example.com/docs/page | grep -i content-typeRun that against your own site after you build it. A content-type of text/html means it is not working, whatever the status code says.
What this does not show
Six sites, one snapshot, documentation entry points only. It does not measure adoption across the web, does not check whether the Markdown is complete or truncated, and does not establish that anything consumes it. It is a sample of what a handful of serious teams have chosen to build.
Common questions
How did you verify this?
Two curl requests per site with the OAI-SearchBot user agent — one to the .md path, one with Accept: text/markdown — reading the returned content type. No rendering, no retries.
Is this a standard?
No. There is no specification for it and no agreed convention on where the Markdown lives. That is why the suffix pattern varies between sites and why a client cannot rely on it.
How does this relate to llms.txt?
They pair naturally: llms.txt says which pages matter, and a Markdown variant makes each of those pages cheap to read. Neither requires the other.
Does it help Google?
No evidence either way, and we would not expect it to. Google renders HTML perfectly well. This is aimed at clients that do not.
The one line to take away
Three of six documentation sites already serve Markdown, and one returns HTML from a .md URL while reporting 200. If you build this, verify by content type — then ship it and push the change for indexing.
