Content signals in robots.txt: 7 of 34 sites have them

RULE CHANGE · 2026-08-15 · Cloudflare Content Signals Policy

Content signals in robots.txt are three extra values — search, ai-input and ai-train — that state what may be done with your content after it has been fetched, rather than who may fetch it. Cloudflare launched the policy on 24 September 2025 and applied it automatically to over 3.8 million domains. We fetched 34 well-known robots.txt files on 15 August 2026 and found it on 7 of them.

What the three signals mean

Everything else in robots.txt governs access. These govern use, which is a different question and one no existing directive could express.

The middle one is the interesting addition. Until this existed there was no way to say "index me, quote me in answers, but do not train on me" — the available tokens bundled those together.

The syntax

One line, comma separated, placed alongside your normal rules.

Content-Signal: search=yes, ai-input=yes, ai-train=no

User-agent: *
Allow: /

Content signals in robots.txt: who actually has them

We fetched /robots.txt from 35 well-known sites with the OAI-SearchBot user agent on 15 August 2026. 34 responded, and 7 carried a Content-Signal line.

Two things in that table are worth stopping on

First, patreon.com carries a fourth signal, use=reference, which is not one of the three Cloudflare documents. Either the vocabulary has grown beyond that announcement or someone is writing their own — and a parser expecting exactly three values would silently skip it.

Second, Cloudflare's own site publishes ai-train=yes, while the policy it deployed to 3.8 million domains sets ai-train=no. That is not a contradiction — the default they applied to others is the cautious one, and they made a different choice for themselves — but it is a good reminder that a default is not a recommendation.

Does anyone have to honour it?

No, and this is the part to be clear about. It is a statement of preference, like the rest of robots.txt, and no crawler operator is obliged to act on it.

What it adds is a legal framing rather than a technical one. The policy text includes the clause "ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF RIGHTS", which points at EU copyright law's reservation mechanism. Whether that holds up anywhere is a question for lawyers, and we are not going to pretend to answer it.

What we cannot tell you

Whether any AI operator changed behaviour because of a content signal. Nobody publishes that, and we have no server logs from a site that added one. We measured what is written in files, not what any crawler did about it.

We also cannot tell how many of those 7 sites chose their values deliberately. Five of them are on Cloudflare's own infrastructure, and the policy was applied automatically to millions of domains — some of these lines may be defaults nobody has read.

Common questions

How did you verify this?

Definitions and the deployment number are quoted from Cloudflare's own announcement. The adoption numbers are our own: one curl per site, one regex for Content-Signal. Raw results are saved alongside this post.

Should I add it?

It costs one line and expresses something no other directive can. But add it knowing that it is a preference with no enforcement, and that it does not replace deciding which crawlers you allow in the first place — that part is still robots.txt for AI crawlers.

Does it work if I am not on Cloudflare?

The line is just text in your robots.txt, so you can write it on any host. Whether it is read is the same open question either way.

The one line to take away

Content signals separate "who may fetch" from "what may be done with it", which is a genuinely new thing to be able to say — and after nearly a year, 7 of 34 well-known sites have said it. Deciding your values, shipping them and pushing the change for indexing takes about five minutes.