Website builder SEO: the five layers a hosted platform decides for you

Website builder SEO is decided by five layers, and a hosted platform hands you full control of one of them, partial control of two, and none of the other two. Knowing which is which is the whole skill: it tells you when a workaround will actually reach the problem, and when you are about to spend a week on something the platform was never going to let you touch. Moving to another platform is the last item in this chapter on purpose.

Read this first

This chapter is about what you cannot change. The companion chapter is about what you can — the diagnostic order, and which layer each platform does open up, are in unblock AI crawlers by platform. Read that one first if your immediate problem is a crawler being refused, because three quarters of those turn out to be fixable.

One definition: a hosted platform here means anything where you do not control the server process — a site builder, a storefront, a managed CMS, a static host. The line that matters is not open source versus proprietary. It is whether you can put arbitrary code in the request path.

Website builder SEO is really a question about five layers

Every complaint that starts with "the platform won't let me" lands in one of these five. The middle column is what a typical hosted platform gives you, and it is the reason the same advice works on one and is impossible on another.

Read that table top to bottom before you plan any fix. Work that lives in the first two rows always ships. Work that depends on the last row cannot be done at all, and no amount of cleverness converts one into the other.

Layer one: the file you do not write

Most hosted platforms generate robots.txt for you. The good ones give you a hook rather than a text box. Shopify's own documentation is a fair example of the shape: the robots.txt.liquid template "isn't included in any themes by default", you create it yourself in the Templates folder, and the docs recommend using the provided Liquid objects rather than plain text because the default rules are "updated regularly to ensure that SEO best practices are always applied".

That recommendation is the trade-off in one sentence. Replace the generated file with your own text and you freeze the platform's defaults on the day you wrote it, then inherit the maintenance forever. Extend it through the hook and you keep the updates, at the price of not being able to remove a rule the vendor considers a best practice.

The substitute when there is no hook at all: page-level directives. A meta name="robots" tag reaches indexing decisions that robots.txt cannot express anyway, since a disallowed page can still be indexed from links. What page-level directives cannot do is stop the fetch, so they are no help when the goal is saving bandwidth.

Layer two: the decision at the edge

This is the layer people most often assume they have and most often do not. Whether a named crawler is served at all is decided in front of your application by rules you did not write, and vendors differ on both the defaults and what they let you override.

Two documented behaviours show the range. Cloudflare describes AI Crawl Control as "Available on all plans" with the ability to "Set allow or block rules for individual crawlers" — full per-crawler control, no upgrade required. Vercel's managed ruleset for AI crawlers is "inactive by default", with log or deny as the two settings, and the list of names is maintained by Vercel rather than by you.

The substitute: put a provider you do control in front of the one you do not. It works, and it has a cost that is easy to miss. Vercel documents that its bot protection "doesn't work when a reverse proxy (e.g. Cloudflare, Azure, or other CDNs) is placed in front of your Vercel deployment". Stacking buys you the outer control and disables part of the inner one. Decide which of the two you actually needed.

Stacking a second provider in front does not add control. It moves the decision, and switches off whatever the inner one was doing.

Layer three: the response headers

Headers are where the platform's opinion is least negotiable and, as it happens, least used. In our own survey of 27 homepages, exactly one sent an X-Robots-Tag at all and twenty sent neither that header nor a meta robots tag — the numbers are in the X-Robots-Tag survey. Freshness headers are similar: 13 of the same 27 send neither a last-modified value nor an ETag, so a conditional request is impossible against half the panel.

The practical reading is reassuring. If your platform will not let you set these, you are in the majority, and the majority includes sites far larger than yours. This is a layer where being locked out costs you an optimisation rather than a capability.

The substitute: for indexing and snippet rules, the meta tag in the head does the same job for HTML pages. For files that have no HTML — a PDF, an image, a data export — there is no substitute, and that is the honest limit. If your business depends on controlling how a PDF is indexed, and your platform will not send a header, that specific requirement is unmet.

Layer four: the log you will never get

You cannot reconstruct a raw request log. Every other item in this chapter has a workaround; this one does not. Without it you cannot say which crawler came, how often, what it asked for, or what it received.

The substitute is weaker and you should know how much weaker. You can measure from outside on a schedule — request your own pages as each user agent, record status and bytes, repeat weekly — and you can read the coverage and performance reports on the search side. Between them you get arrivals you inferred and outcomes you observed. What you do not get is the middle: attempts that failed, and clients that never identified themselves.

Every field test we publish is shaped by this same gap, which is why each one carries a sample size and a date instead of a claim about behaviour over time. If you are locked out of logs, write your conclusions the same way, and do not let a weekly snapshot get quoted back to you as a trend.

Layer five: the template around your words

The body is yours everywhere. That sounds like a consolation prize and it is actually the largest lever in the list, because the things that decide whether a page can be quoted live in the body: the heading that matches a real question, the self-contained paragraph underneath it, the table instead of four sentences of prose.

Where hosted platforms genuinely bite is the head. Some give you a free-form injection slot, some give you named fields and nothing else. If yours is the second kind, structured data is the first casualty, since it needs a script element you may have no way to place.

The substitute: check whether the platform emits its own structured data before concluding you need to add any. Several site builders and storefronts do, and a second block fighting the first is worse than neither. If the platform emits nothing and gives you no slot, that is a real gap — and it affects a narrower set of outcomes than most people assume, which is a good reason not to let it drive a migration on its own.

The deliverable: score your platform in fifteen minutes

Fill this in once and keep it. It stops the recurring argument about whether something is worth attempting, and it stays valid until you change platform.

Any row that comes back "No" is a constraint to design around, not a defect to fight. The point of writing them down is that next quarter somebody will propose a project that depends on one of them, and this sheet answers in a minute what would otherwise take a week to discover.

When moving is the right call, and when it is not

Moving a site is expensive in ways that do not show up in the estimate: every URL, every internal link, every piece of structured data, and a recovery period during which your numbers are not comparable to anything. It is worth it when a closed layer blocks something central to how you make money, and it is a bad trade when it is driven by a single missing feature.

Three signals that the constraint is real rather than annoying: it blocks a revenue path and not a metric; you have already implemented the substitute and measured that it falls short; and the same constraint has come up more than twice from different directions. Fewer than three and the honest answer is usually to write the constraint into the platform scorecard and move on.

What does not justify a move: an SEO checklist item with no traffic attached to it, a header you would set once and never look at, or advice written for a stack you are not on. Making the workaround for a closed layer a repeatable job rather than a research project is what QueryWin is being built to do.

Common questions

Is a website builder bad for SEO?

Not in the layer that matters most. Headings, answers and page structure are editable everywhere, and those decide whether a page can be quoted. What builders restrict is headers, edge rules and logs, which cost you optimisations rather than the ability to compete.

Can I edit robots.txt on a hosted platform?

Sometimes, and often only through a template hook rather than a text file. Shopify, for instance, expects a robots.txt.liquid template you create yourself, and recommends extending the generated rules instead of replacing them.

What if my platform will not let me add structured data?

Check what it already emits first, since a duplicate block is worse than none. If it emits nothing and offers no injection slot, treat it as a genuine gap, and weigh it against everything else before letting it decide a migration.

How do I track crawlers without server logs?

Measure from outside on a fixed schedule and pair it with the search console reports. You will have arrivals you inferred rather than arrivals you observed, so state a sample size and a date on every number and never call a snapshot a trend.