Newest first · RSS
Field notes
Rule changes at AI engines and platforms, experiments measured on live sites, and release notes.
Crawling & IndexingServing markdown to AI crawlers: 3 of 6 docs sites already do
Anthropic, Next.js and Cloudflare return text/markdown when you append .md to a docs URL. One site returns HTML from its .md path while reporting 200 — check the content type, not the status code.
Crawling & IndexingCloudflare AI Crawl Control: per-crawler visibility you may already have
Cloudflare AI Crawl Control shows which AI services fetched your pages and lets you allow or block each one. Available on all plans, zero configuration — and most site owners have never opened it.
Crawling & IndexingNaming a crawler usually means blocking it: 82 of 99 groups
Of 99 named AI crawler groups across 14 sites, 82 were a full-site Disallow. But three sites named crawlers and blocked none — counting names conflates two opposite decisions.
Crawling & IndexingContent signals in robots.txt: 7 of 34 sites have them
Content signals in robots.txt state what may be done with your content after it is fetched. Cloudflare launched them in September 2025 and applied them to 3.8 million domains. We found them on 7 of 34 sites.
Crawling & IndexingMost homepages carry no structured data: 9 of 14
Of 14 well-known homepages fetched as an AI crawler, 9 delivered zero JSON-LD blocks. The bar is lower than the volume of schema advice suggests.
Crawling & IndexingOne company, several crawlers: blocking the wrong one costs you
Every major AI company now runs several crawlers with different jobs. Blocking GPTBot does not remove you from ChatGPT — that is OAI-SearchBot. Here is the map from each operator's own docs.
Crawling & IndexingYour docs are emptier than your homepage: 7 of 10 sites
On 7 of 10 well-known sites, the documentation page delivered less readable text to an AI crawler than the marketing homepage. On linear.app the docs returned an eighth as much.
Crawling & IndexingThere is no noai directive — nosnippet is the real AI lever
There is no noai directive in Google's documentation. The tag that controls what AI Overviews and AI Mode may quote is max-snippet, and Google names those surfaces explicitly.
Crawling & IndexingSchema markup for AI search: which types are still live
Schema markup for AI search is worth adding, but two of the most-recommended types no longer produce rich results. Here is what Google still supports, four blocks to paste, and the policy line that gets sites penalised.
Crawling & IndexingWho blocks AI crawlers and who publishes llms.txt: 30 sites, two opposite strategies
We fetched robots.txt and llms.txt from 30 well-known sites. News publishers name AI crawlers by the dozen and publish no llms.txt; developer-tool companies do the exact opposite.
Crawling & Indexingllms.txt: what it is, a template, and what it cannot do
An llms.txt lists your most useful pages so an agent can find them without crawling everything. Here is the format, a template, who actually publishes one, and the five things it does not do.
Crawling & IndexingWhat AI crawlers get from JavaScript sites: 342 to 25,082 characters
What AI crawlers get from JavaScript sites varies 73-fold. We fetched 15 well-known homepages as an AI crawler: 14 returned 200, and the readable text ranged from 342 to 25,082 characters.