Cloudflare AI Crawl Control: per-crawler visibility you may already have
Cloudflare AI Crawl Control shows which AI services fetched your pages and lets you allow or block each one. Available on all plans, zero configuration — and most site owners have never opened it.
RULE CHANGE · 2026-08-15 · Cloudflare documentation
Cloudflare AI Crawl Control is a dashboard that shows which AI services fetched your pages and lets you allow or block each one, and it is available on all plans with zero configuration. That combination matters more than the feature list: if your site is behind Cloudflare, you already have per-crawler visibility that most site owners assume requires server logs they cannot get.
What Cloudflare AI Crawl Control does, in its own words
Four capabilities, quoted from the product documentation as read on 15 August 2026.
| Capability | Documented as |
|---|---|
| Visibility | "See which AI services access your content" |
| Control | "Control access with granular policies" — allow or block individual crawlers |
| Analysis | "Analyze AI traffic" to understand crawler interaction patterns |
| Compliance | "Monitor robots.txt compliance" and "Track which crawlers follow your directives" |
Availability is stated plainly: "Available on all plans", and it "Deploy[s] with zero configuration".
Why the compliance monitoring is the interesting part
Everything in robots.txt is a request. Whether a crawler honours it has, until recently, been unobservable to anyone without server logs — and most people running a site on a managed platform never see those.
A feature that tracks which crawlers follow your directives closes exactly that gap. It is the difference between "we asked them not to" and "we know whether they did", and that distinction runs through every chapter of our handbook.
Pay per crawl is real but not available
The documentation lists a monetisation option that would "Allow AI crawlers to access content by paying per crawl". Its status is given as "private beta".
We mention it because it will be quoted at you as though it were shipping. It is not something you can turn on today, and a plan that depends on it is a plan with a dependency on someone else's beta.
What this does not mean
Cloudflare AI Crawl Control being available does not make it a measurement of the whole internet, and it does not tell you anything about crawlers that never reach Cloudflare because they were stopped elsewhere. It sees traffic arriving at Cloudflare's edge, which is a large and specific window, not a universal one.
We also have not used it against a live property with meaningful AI crawler traffic, so we cannot tell you how granular the data actually is once you open it. Everything above is what the documentation says, not what we observed.
Where this sits relative to the rest of your setup
It does not replace anything. It observes and enforces at the edge, which is a different layer from the files you write.
| Layer | What it is |
|---|---|
| robots.txt | A request. Crawlers choose whether to obey. |
| Content signals | A statement about permitted use, not access. |
| Edge controls like this one | Enforcement. The request either completes or it does not. |
The order matters when something goes wrong: a crawler getting a 403 is being stopped by the third row, and no amount of editing the first row will change it.
Common questions
How did you verify this?
By reading Cloudflare's AI Crawl Control documentation on 15 August 2026 and quoting the capability descriptions, the availability statement and the beta status directly. We did not test the product.
I am not on Cloudflare. What is the equivalent?
Server logs, if you can get them. That is the underlying data source either way — the product is a view over it. If your host gives you no log access, you have no equivalent, which is itself worth knowing before you choose a host.
Does using it change anything about my robots.txt?
No. They are separate layers, and both are worth getting right. Which crawlers to name and what to say about them is robots.txt for AI crawlers.
Should I block crawlers at the edge instead of in robots.txt?
Only if you mean it. An edge block is enforcement, so it also stops the well-behaved crawlers that would have honoured a polite request — including the ones that put you in answers.
The one line to take away
If you are behind Cloudflare, per-crawler visibility is already switched on and most site owners have never opened it. Looking, deciding, and then pushing the resulting changes for indexing is a better first move than adding another line to a file nobody enforces.
