Faceted navigation: which filter URLs to let a crawler have

Faceted navigation earns crawling one filter at a time, never as a whole feature. Keep the filters people actually search for, close the rest in robots.txt, and let empty combinations return a real 404. Google's reasoning is the useful part: a crawler cannot tell a useful filter URL from a worthless one without fetching it first, so the sorting has to be done by you, in advance. The decision table and a tested pattern set are below.

Before you start

This chapter sits one step after a different question — whether crawling volume is a problem on your site at all. Sites with a few hundred pages and no filters usually answer no, and that case is worked through in is crawl budget your problem. Two mechanisms appear here as alternatives rather than as the main route, and each has its own chapter: how to use canonical tags and when to noindex a page. Read this one when Search Console keeps reporting addresses nobody on your team wrote by hand.

Why faceted navigation URLs multiply

Every filter the interface offers is another address, and the addresses combine. Google's definition is deliberately plain: faceted navigation "allows its visitors to change how items (for example, products, articles, or events) are displayed on a page". Three filters with ten values each describe a thousand combinations of one shelf of products, and the shelf itself only ever needed one page.

The part that matters for this decision is what a crawler does when it meets those addresses. From the same page: "Because the URLs created for the faceted navigation seem to be novel and crawlers can't determine whether the URLs are going to be useful without crawling first, the crawlers will typically access a very large number of faceted navigation URLs" (Faceted navigation best practices, read 2026-09-06). The cost is stated in the next breath: "If crawling is spent on useless URLs, the crawlers have less time to spend on new, useful URLs."

There is a second cost, and it lands on your servers rather than on Google's. The same document notes that crawling these URLs "tends to cost sites large amounts of computing resources due to the sheer amount of URLs and operations". Each request is a listing your application has to assemble from scratch: a filtered read, a render, a cache entry no human will ever ask for again.

A filter URL earns a crawl only if somebody searches for the thing it filters to. Everything else is a combination you invented.

That sentence cuts in both directions, which is what makes this a judgement call rather than a rule you can copy. Block everything with a parameter and you lose the filtered pages that carry real demand — a brand page, a size that people search for by name. Block nothing and requests go to combinations no visitor has ever typed. Neither failure announces itself.

The four mechanisms, and which of them stops the fetch

Google documents four ways to deal with these URLs, and they are not interchangeable. Two prevent the request. Two are instructions carried inside a page that has to be fetched before the instruction can be read. If your goal is to reduce crawling, only the first two are aimed at it.

The fragment route works because of a documented limit rather than a feature. "Google Search generally doesn't support URL fragments in crawling and indexing", so a filter expressed after a hash "will have no impact on crawling (positive or negative)". That is a clean outcome if your filters are already rendered in the browser, and a rewrite of the whole listing interface if they are not.

The canonical route is the one people reach for first and the one that disappoints most often. Pointing filtered variants at the unfiltered listing "may, over time, decrease the crawl volume" of those variants. Read the hedge in that sentence: may, over time. The URL still has to be fetched for the tag to be seen at all, so the request you wanted to avoid has already happened.

The nofollow route has a failure mode built into its requirement. Google's caveat is that "every anchor pointing to a specific URL must have the rel='nofollow' attribute". One filter link in a footer, one in an internal search result, one in a sitemap someone generated last year, and the attribute stops doing anything. On a site with a filter interface, that condition is hard to hold and harder to keep true.

Do it: sort your filters before you write a rule

Five steps. Each has a finish line you can check the same day, and the fourth is the one that keeps you from blocking your own category pages.

1. List every parameter your filter interface can emit. Read it out of the code or out of a crawl of one category page — not out of memory, because filter interfaces grow parameters nobody documented. Finish line: a written list of parameter names.
2. Mark each parameter as keep or close, based on one test: does anybody search for the thing this filter narrows to? Finish line: every name on the list carries a verdict and a one-line reason.
3. Decide what happens when two filters combine. The default answer is close, because demand for a combination is rarer than demand for either half. Finish line: one written rule covering combinations, so it does not get re-argued per parameter.
4. Write the robots.txt rules, then run your own URLs through them before shipping. Finish line: the base listing and plain pagination come back allowed, and every address on the close list comes back blocked.
5. Make empty combinations return the right status. Google is explicit here: "Return an HTTP 404 status code when a filter combination doesn't return results." For a single-page app where a real status code is not available, Google points to its JavaScript SEO guidance instead. Finish line: a combination with zero products answers 404, not an empty listing with a 200.

Deliverable: the decision table and the robots.txt patterns

First, which kind of filter URL gets which treatment. The right-hand column is the part that makes this a decision instead of a recipe — every row costs you something, and the cost is the reason you cannot copy someone else's robots.txt.

Second, the patterns. This set closes parameters by default and reopens the two that earn a crawl, which is the shape most listing sites end up with. It uses only what robots.txt actually supports: * for any sequence, $ for end of address, longest matching rule wins, and allow beats disallow on a tie.

# robots.txt — close parameters by default, reopen what earns a crawl.
User-agent: *

# 1. Every URL carrying a parameter is closed.
Disallow: /*?

# 2. Plain pagination stays open; pagination plus a filter does not.
Allow: /*?page=
Disallow: /*?page=*&

# 3. One filter with real demand stays open; the same filter
#    combined with anything else does not.
Allow: /*?brand=
Disallow: /*?brand=*&

# Verified against these, longest-match-wins:
#   /shoes                                    base listing         allowed
#   /products/acme-runner                     product page         allowed
#   /shoes?page=3                             plain pagination     allowed
#   /shoes?brand=acme                         filter with demand   allowed
#   /shoes?brand=acme&color=green              two filters          blocked
#   /shoes?page=3&color=green                  pagination + filter  blocked
#   /shoes?sort=price_asc                     sort order           blocked
#   /shoes?products=fish&color=radioactive_green&size=tiny     blocked
#   /shoes?color=green&brand=acme              brand not first      blocked  <-- see below

The last line is the reason to test rather than to trust. The reopened filter only survives when it is the first parameter, because the allow pattern anchors on the character that opens the parameter string. Put the same filter in second position and the broad disallow catches it. That is not a flaw to work around; it is the mechanical reason Google's own best practices ask you to "keep the filter order in the URL consistent and logical". Pick one order, emit it everywhere, and the rules stay predictable.

Three more things belong in the same file review. The base listing must stay allowed — check it explicitly, because a pattern written to catch a parameter can match a path segment by accident. Use standard & separators rather than commas, semicolons or brackets, which is Google's stated preference for filter addresses you do intend to keep. And remember that robots.txt user agent tokens are coarser than people expect: Google states that Googlebot Smartphone and Googlebot Desktop "obey the same product token (user agent token) in robots.txt, and so you cannot selectively target either" one (Googlebot, read 2026-09-06).

Confirm it twice. Before shipping, run every URL shape you care about through a matcher that implements longest-match-wins, or paste them one at a time into the robots.txt report in Search Console. After shipping, the crawl stats report is where a change in request volume would appear, and the page indexing report is where filtered addresses should stop arriving as newly discovered URLs. We are not going to tell you how long that takes, because we have not measured it.

Three ways this goes wrong

The expensive one is blocking the listing you were trying to protect. A rule written loosely enough to catch a parameter can also match a category path that happens to contain the same string, and the listing then stops being crawled along with its filters. Nothing errors. You find out weeks later, from a report rather than from the site. The check is one line long: run the base listing address through your own rules and confirm it comes back allowed.

The second is reaching for canonical or nofollow when the goal was to stop the fetch. Both are instructions about a URL that has already been requested, which means they cannot reduce the requests. Canonical may reduce crawling of the variants over time, on Google's own hedged wording; nofollow needs an attribute on every anchor that points at the address, everywhere on the site, forever. If the problem you are solving is request volume, those two are the wrong drawer.

The third is closing a filter that was already earning. Filtered listings for a brand, a size or a material sometimes carry demand of their own, and those pages had impressions before you wrote the rule. Blocking them costs you the traffic and gives back very little, since one pattern is a small share of the crawling. Look at the search performance data for filtered addresses first. Deciding afterwards means comparing against a number you no longer have.

Where this stops

Four limits. The first two are about what nobody knows, and the last two are about what you can reach on your own platform.

1. Google publishes the mechanisms and the caveats. It does not publish a threshold for how many filter URLs is too many. There is no number to compare yourself against, and we do not have one either — anybody quoting you a ratio of filter URLs to product pages made it up.
2. We have not run a controlled test that isolates the effect of blocking filter URLs on anything downstream. On a live site nobody can, because you would need a second identical site to compare against. What is documented is the mechanism, and the mechanism is a reason to act, not a promise of an outcome.
3. On a hosted platform you may not have all four mechanisms available. Editing robots.txt is possible on some builders and not others; rewriting every filter link into a fragment needs template access; returning a 404 for an empty combination is decided by the platform, not by you. Check which of the four you can actually reach before designing around one you cannot.
4. Robots.txt governs fetching, not what is already in the index. If addresses are already indexed and you want them gone, that is a different job with a different tool, and closing the path to them first makes it slower rather than faster.

One more thing worth checking before any of this: whether crawlers reach your listing pages at all. Filtering the crawl is pointless if the base pages are being refused a layer above your application, which is what the AI crawler access check is for.

Common questions

Does faceted navigation hurt SEO?

Not by existing. The documented problem is that crawlers access a very large number of these addresses because they cannot judge usefulness without fetching, and that crawling spent on useless URLs is crawling not spent on useful ones. A filter interface with a handful of parameters on a small site is not the situation that describes.

Should I block filter URLs in robots.txt or use canonical?

Robots.txt, if the thing you want to stop is the request. Canonical is read from inside a page that has already been fetched, and Google's own wording is that it may decrease crawl volume over time. Canonical is the better tool when you want the filtered page to exist and consolidate, and the worse tool when you want it never to be requested.

How do I handle faceted navigation on Shopify?

Find out first which of the four mechanisms your plan and theme actually expose. Some hosted platforms allow a robots.txt edit through a template, some do not, and the filter link markup is often generated by the theme rather than by you. Where robots.txt is not editable, the fragment route and the canonical route are what remain, and they answer different questions.

Will blocking filter URLs remove my category pages from Google?

Only if your patterns match those pages, which is exactly why step four exists. Run the base listing and one product address through your rules before shipping and confirm both come back allowed. The base listing must stay open.

What should a filter combination with no results return?

A 404. Google's best practices ask for "an HTTP 404 status code when a filter combination doesn't return results", which keeps empty listings out of the crawling pool without a robots.txt rule for every impossible pair. Where a real status code is impossible, Google's JavaScript SEO guidance is the documented fallback.