Unblock AI crawlers: which layer your platform actually lets you edit

To unblock AI crawlers you have to fix the layer that is actually refusing them, and on a hosted platform that layer is usually not the one you can edit. Four of the platforms in this chapter decide it at the edge or inside a template, and each one gives you a different amount of room. This chapter gives you the order to diagnose in, what each platform actually controls, and the two places where a fix silently does nothing.

Read this first

You need a reading of what happens today before you change anything. If you have not run the per-user-agent check yet, do that first — the response codes it gives you are the input to every decision below, and guessing which layer is at fault wastes an afternoon. The mechanism itself, and how to tell a CDN block from an origin block by the response headers, is covered in CDN blocking AI crawlers.

One definition, so the rest holds: an AI crawler is a named automated client that fetches your pages on behalf of an AI product. Which of them are worth letting in, and which names are control tokens rather than crawlers, is decided in allow the right AI crawlers. This chapter assumes you have already decided who you want in, and answers only how to actually let them in.

Why a 200 in your browser proves nothing

Three systems get a vote before a crawler reaches your HTML: the robots.txt file you publish, the edge rules of whatever sits in front of your site, and the origin itself. Your browser passes all three because it looks like a browser and comes from a residential address. A named crawler often fails at the second one, and the second one is the layer most hosted platforms keep for themselves.

That split explains the most common wasted fix. Somebody edits robots.txt, deploys, checks the file in a browser, sees the new rules, and concludes the site is open. The file is a request. The edge rule is the decision, and nobody edited it.

robots.txt is a request. The edge rule is the decision. On a hosted platform you rarely get to edit both.

Diagnose in this order

Work outside in. Each step tells you whether the next one is even relevant, and stopping at the first refusal saves you from changing three things at once.

1. Fetch your homepage with a named crawler user agent and record the status code and the response headers. A 403 or a challenge page is an edge decision; a 200 moves you to step two.
2. Fetch your /robots.txt and read the group that applies to that name. A blanket Disallow: / under a wildcard group applies to every crawler that has no group of its own.
3. Look in the delivered HTML for a meta name="robots" with noindex, and in the response headers for X-Robots-Tag. These do not stop the fetch, so they never show as a failed request.
4. If all three are clean and the crawler still gets nothing useful, the problem is what the page contains rather than whether it was served.

Steps one and three catch different failures and are easy to confuse. Step one answers "was I served"; step three answers "was I allowed to be used". A page can pass one and fail the other with no error anywhere.

How to unblock AI crawlers on four hosted platforms

Each row below is the layer that platform actually lets you edit, taken from its own documentation on 2026-08-21. The differences are not cosmetic — they decide what a fix can even do.

Cloudflare's row is the good case: one screen, one crawler at a time, and it is on every plan rather than gated behind an upgrade. If Cloudflare is in front of your site, start there before you touch anything in your repository.

Vercel is the row people misread most often. Its managed ruleset for AI crawlers is off by default, which means an unexplained block on a Vercel site is usually not that ruleset. Its separate bot-protection ruleset is also off by default and, when on, "automatically excludes verified bots, such as Google's crawler, from evaluation" — so a search crawler and a newer AI crawler can get opposite treatment from the same switch. Vercel also documents a trap for anyone stacking providers: "Bot Protection doesn't work when a reverse proxy (e.g. Cloudflare, Azure, or other CDNs) is placed in front of your Vercel deployment."

WordPress is the one where the wording of the setting misleads. The checkbox is called "Discourage search engines from indexing this site" and its own documentation is explicit about the limit: "Neither of these options blocks access to your site — it is up to search engines to honor your request." Since version 5.3 it works by writing a meta tag, not by editing robots.txt, so anyone auditing only the file will find nothing wrong.

Shopify hands you the file but asks you not to rewrite it from scratch. Its documentation recommends the Liquid objects over plain text because the default rules "are updated regularly to ensure that SEO best practices are always applied" — a hand-written replacement freezes those defaults on the day you wrote it.

Two platforms this chapter cannot cover today

Webflow's help centre returned 403 to us on 2026-08-21 and Netlify's indexing page returned 404, so neither could be quoted from its own documentation. We are not going to describe a settings screen we could not open. If you are on one of those two, run the four diagnostic steps above, find which layer refuses, and then read that vendor's current page for that specific layer.

This is the general shape of the boundary. A chapter can tell you which layer to look at and what a fix at that layer can do; only the vendor's current documentation can tell you where today's toggle lives, and those move.

The deliverable: one pass, seven lines

Copy this into your notes and fill the right-hand column. It takes about fifteen minutes and it survives a platform migration, because every row is a layer rather than a menu path.

Re-run the first row after every change. It is the only line in the table that reports what a crawler actually experiences; the other six report what you configured, and those two are not the same thing.

Some rows will come back with no control at all on your platform, and that is a different problem with a different answer — the five layers a hosted platform keeps for itself, and the substitute for each, are in website builder SEO limits.

What goes wrong

Three failures account for most of the wasted effort here, and none of them produce an error message.

1. Fixing the file when the edge is refusing. The symptom is a correct robots.txt and an unchanged 403. Re-run the first diagnostic row after the deploy; if the status did not move, the file was never the reason.
2. Unblocking the fetch and leaving a noindex. The crawler is served, the page is not usable, and every request log looks healthy. Only step three catches it.
3. Two providers stacked, each assuming the other decides. Vercel's documentation states its bot protection does not work behind a reverse proxy. Whichever system is on the outside is the one making the call, and it is not always the one you configured.

When you want to know what a specific crawler receives right now rather than what your settings say, run it through the AI crawler accessibility check and compare the result against the first row of the table above.

Common questions

How do I unblock AI crawlers on my site?

Find the refusing layer first — edge, robots.txt, or page directive — then change only that layer. On Cloudflare and Vercel that is a dashboard rule; on WordPress it is a setting that writes a meta tag; on Shopify it is a Liquid template you create yourself.

Is Cloudflare blocking AI crawlers by default?

Cloudflare documents AI Crawl Control as available on all plans and working automatically, with allow or block rules set per crawler. What the current default action is for your zone is visible in that dashboard, and that is the only reliable place to read it.

I changed robots.txt and nothing happened. Why?

Because robots.txt is a request that a crawler chooses to honour, and it has no effect at all on a request that is being refused before it reaches your site. If the status code did not change, the block is above the file.

Does a noindex tag stop a crawler from fetching the page?

No. The fetch succeeds and the page is retrieved; the directive tells the engine not to use it. That is why a site can look perfectly reachable in a log and still be absent from results.