llms.txt: what it is, a template, and what it cannot do

An llms.txt is a markdown file at your site root that lists your most useful pages with one line of description each, so an agent can find what it needs without crawling your whole site. It is a proposal, not a standard anyone is obliged to honour, and no major engine has publicly committed to reading it. It is also cheap to publish and increasingly audited for — which is the honest case for doing it.

Read this first

Publish this after your pages actually reach crawlers. An llms.txt pointing at URLs that return a shell to a crawler just routes agents more efficiently to nothing — see can AI crawl JavaScript if you have not checked that yet.

What llms.txt is, precisely

It is a markdown file, placed at /llms.txt or at any path to cover the pages beneath it, containing a short description of the site and curated links with one-line summaries. The proposal was written by Jeremy Howard in September 2024 and is now at v2, revised in August 2026.

The reasoning behind it is stated plainly in the proposal: web pages are built for people, an HTML page "wraps its information in navigation, ads, and JavaScript", and converting that back into clean text is "difficult and imprecise". The file is an attempt to hand over the clean version directly.

A sitemap tells a machine what exists. An llms.txt tells it what matters.

The format, and a real one

Four elements, in this order: an h1 with the name, a blockquote summary, then h2 sections each containing a list of links with a colon and a description. Here is the opening of Stripe's, fetched on 15 August 2026:

# Stripe

> Stripe is a technology company that provides financial infrastructure
> for businesses...

## Payments

- [Stripe Payments](https://stripe.com/payments): Capture more revenue with
  a unified payments solution...
- [Payment methods](https://stripe.com/payments/payment-methods): Convert
  more customers...

Note what it is not. It is not a dump of every URL, it is not marketing copy, and the descriptions are written to help someone decide whether to open the link. That editorial judgement is the entire value of the file — a machine-generated list of every page is a sitemap with extra steps.

A template you can fill in

Start with fifteen to thirty links. If you cannot describe a page usefully in one line, it probably does not belong in the file.

# Your Company

> One or two sentences: what you do, for whom, and what makes
> your documentation worth reading.

## Getting started

- [Quickstart](https://example.com/docs/quickstart): Set up and make
  your first request in about ten minutes.
- [Installation](https://example.com/docs/install): Supported platforms
  and version requirements.

## Reference

- [API reference](https://example.com/docs/api): Every endpoint, with
  request and response examples.

## About

- [Pricing](https://example.com/pricing): Plans, limits, and what counts
  toward each limit.

Who actually publishes one

We fetched /llms.txt from 30 well-known sites on 15 August 2026. Fourteen served a real one. The split was not random: developer-tool and SaaS companies had them almost universally, and news publishers had none at all.

The proposal itself notes that thousands of sites now publish one, that documentation platforms generate them automatically, and that the AI labs publish llms.txt files for their own developer documentation. The full site-by-site numbers are in who blocks AI crawlers and who publishes llms.txt.

🚫 What it does not do

This is the part most write-ups skip, and it is where the disappointment comes from.

Does anything actually read it?

We do not know, and we are not going to imply otherwise. What can be verified is narrower and still meaningful: the proposal is at v2 after two years of adoption, documentation platforms generate the file automatically, the major AI labs publish one for their own docs, and Chrome's Lighthouse now audits sites for it as part of its agentic browsing checks.

None of that is a crawler operator saying "we read this". The measurement that would settle it is server logs — counting requests to /llms.txt and which user agents made them. That is a study we have not run, because it needs log access we do not have, and we would rather say so than quote someone else's estimate.

How to publish it

The file is static text, so this is a five-minute job on most stacks.

1. Write it, starting from the template above. Curate rather than enumerate.
2. Serve it at /llms.txt with content type text/plain or text/markdown. Put it wherever your static files live.
3. Fetch it back as a crawler and confirm you get the file, not your homepage. Single-page apps commonly return the app shell for unknown paths, which looks like success and is not.

curl -s -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot)" \
  https://example.com/llms.txt | head -20

If that prints HTML, your file is not being served and the 200 is your app's catch-all route answering.

What to put in it, and what to leave out

The hard part is not the syntax, it is deciding what earns a line. A useful test: would you send this link to a customer who asked a specific question? If the honest answer is "not on its own", it does not belong near the top of the file.

In practice the pages that pay off are the ones that answer a question completely — a quickstart, a pricing page with the limits spelled out, a reference section, a troubleshooting page for the failure people actually hit. The pages that waste a line are category listings, anything whose value is a link to somewhere else, and pages that exist mainly to rank.

Keeping it accurate

A stale llms.txt is worse than none, because it confidently points at URLs that have moved. The file is a curated list, which means it goes out of date exactly as fast as your documentation does — and unlike a sitemap, nothing regenerates it for you unless your docs platform does.

The cheapest maintenance is to check the links on the same schedule you check anything else, and to re-read the descriptions whenever a product changes. A short loop over the URLs in the file, checking each returns 200, catches the common failure in seconds.

Three ways this goes wrong

The first is the one we see most, and it is invisible without the check above.

1. The catch-all route answers instead of the file. Your framework returns the homepage for any unmatched path, so /llms.txt returns 200 and HTML. Everything looks configured and nothing is.
2. Listing every page you have. The value is in the selection. A file with 400 links has made no decisions and helps nobody choose.
3. Writing descriptions as marketing copy. "The best solution for modern teams" tells an agent nothing about when to open the link. Say what is on the page.

Common questions

Is llms.txt an official standard?

No. It is a proposal, currently at v2. Adoption is real and growing, and the AI labs publish one for their own documentation, but no standards body has ratified it and no engine is obliged to read it.

Should I publish one if I am not a documentation site?

If you have a set of pages you would genuinely want an agent to read first, yes — the cost is one static file. If your site is ten marketing pages, the file adds nothing a sitemap does not already cover.

Does it help Google rankings?

There is no evidence it does, and we would treat any claim that it does with suspicion. Judge it as a way to guide agents, not as a ranking tactic.

Where do I put it if my docs live on a subdomain?

Serve one per host. The file covers the pages beneath its own path, so documentation on its own subdomain wants its own file at that subdomain's root.

Next step

With the file published and verified, the remaining question in this part of the handbook is whether the pages it points at are structured so an engine can lift an answer out of them — because a well-curated list of unquotable pages is still a list of unquotable pages.

Writing the file takes an afternoon. Keeping it accurate as the site changes, and pushing updates for indexing when it does, is the part that stalls.