When to noindex a page, and why disallow does the opposite
noindex keeps a page reachable and out of search results — but only if the crawler is allowed to fetch it. Blocking the URL in robots.txt means the rule is never read. Here is the four-tool decision table, both implementations, and the debug order that starts with robots.txt.
Use noindex when a page must stay reachable but should not appear in search results, and use nothing at all when you are only trying to tidy up. The rule works only if the crawler is allowed to fetch the page and read it, which is why blocking a URL in robots.txt does not hide it — it prevents the instruction from ever being seen. Most pages people want to hide should be left alone.
Before you start
You need to be able to add a <meta> tag to a page's <head>, or to set a response header. You also need a specific reason. "It does not get traffic" is not one; a page with no traffic costs nothing and removing it from the index gains nothing. This chapter is about the cases where the page's presence is itself the problem.
Why noindex and robots.txt do opposite things
They sound like two strengths of the same idea. They are two different mechanisms, and using them together cancels the one you wanted. Google's noindex documentation, read on 2026-08-24, states the dependency directly:
"For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler."
The same paragraph spells out the failure: "If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it."
Read that twice, because it inverts the intuition. Disallow is a crawling instruction; it says do not fetch. noindex is an indexing instruction; it lives inside the thing you just told the crawler not to fetch. A URL that is disallowed and linked from elsewhere can be listed with no description at all — the worst of both outcomes, since the page is invisible to you and visible to everyone else.
Google also notes that writing the rule in the robots.txt file itself does not work: "Specifying the noindex rule in the robots.txt file is not supported by Google."
The four tools, and what each one actually does
Almost every "how do I get this page out of search" question is answered by picking correctly from this table rather than by any implementation detail.
| Tool | What it does | Use when |
|---|---|---|
noindex | Drops the page from results; page still crawled | Page must work, must not rank |
rel="canonical" | Merges signals onto another URL | Duplicate of a page you keep |
Disallow | Stops the fetch, not the listing | Crawl cost, not visibility |
| Nothing | Google ranks it or ignores it | Thin but harmless pages |
The second row is the one that catches people mid-cleanup. Google's canonicalization guidance says outright that it does not recommend "using noindex to prevent selection of a canonical page within a single site, because it will completely block the page from Search". If two URLs hold the same content and you noindex one, you have not merged them. You have deleted one from search and kept its duplicate content problem.
Which pages are worth the rule
A short list, because it should be short. Everything here shares one property: the page has to keep working for a human, and its presence in search results creates a real problem rather than an aesthetic one.
- Internal search result pages. They generate unlimited URLs and answer nothing on their own.
- Thank-you and order-confirmation pages, which are only meaningful after an action.
- Staging or preview URLs that must be publicly reachable for a client to review.
- Paginated filter combinations that a canonical cannot cover because each one is genuinely a different set.
- Gated assets whose landing page you want ranked and whose file you do not.
Notice what is missing: old blog posts, thin pages, and low-traffic pages. Removing those from the index does not redistribute anything to the rest of the site. If the concern is that they compete with a better page, that is a canonical or a merge, not a noindex.
The deliverable: two implementations and a debug order
The meta tag form, in the <head>. The first applies to every engine that supports the rule; the second targets Google only:
<meta name="robots" content="noindex">
<meta name="googlebot" content="noindex">The header form. This is the only option for a PDF, a video or an image, since those have no <head> to put a tag in:
HTTP/1.1 200 OK
X-Robots-Tag: noindexGoogle's documentation confirms the two are equivalent — "They have the same effect; choose the method that is more convenient for your site and appropriate for the content type" — and that the header exists for "non-HTML resources, such as PDFs, video files, and image files". You can also combine rules, as in <meta name="robots" content="noindex, nofollow" />.
When the page is still in results a week later, work through these in order. The order matters: three of the four are cheaper to check than the one everyone checks first.
- Is the URL disallowed in robots.txt? If yes, the crawler never saw the rule. Unblock it first.
- Has the page been recrawled since you added the rule? Google's own note is that "depending on the importance of the page on the internet, it may take months for Googlebot to revisit a page".
- Does Googlebot actually receive the tag? Use the URL Inspection tool to see the HTML Googlebot got, not the HTML your browser got.
- Only then suspect the implementation.
Step 3 is where a client-rendered site usually fails: the tag exists in the DOM after JavaScript runs and not in the served HTML. You can see the difference yourself with a check of what crawlers receive from the page.
The expensive failure is the noindex nobody meant to ship
Every rule in this chapter is reversible except this one, and it is reversible only after you notice. A staging environment is usually configured to keep itself out of search, and that configuration lives in the same template, theme or environment variable as production. When the two get merged, or when a content system's "discourage search engines" switch survives a launch, the whole site goes out with the rule attached.
It is quiet in a way most mistakes are not. The site is up. Pages render. Nothing errors. Traffic simply never arrives, and because a new site has no baseline, the absence looks like the normal slow start of a new domain. Weeks of it look plausible.
The check costs one request against your own homepage, and it is worth putting into whatever list you run after a deploy.
curl -sIL https://example.com/ | grep -i 'x-robots-tag'
curl -sL https://example.com/ | grep -io '<meta[^>]*robots[^>]*>'If either line prints noindex on a page you want ranked, stop reading and fix that. Search Console's Page Indexing report names this state explicitly — "Excluded by 'noindex' tag" — so a site with any Search Console history will tell you as well, a day or two later. The command is faster and works before the property is verified.
What noindex will not do for you
It is not a privacy control. The page stays public, stays linked, and stays fetchable by anything that ignores the rule. Google's documentation is careful to scope its own promise: "some search engines might interpret the noindex rule differently. As a result, it is possible that your page might still appear in results from other search engines." Anything that must not be read needs authentication, not a meta tag.
It is also not fast. The rule is applied on the next crawl, and Google's guidance for urgent cases points at the removals tool instead. If you need something gone today, noindex is the wrong instrument.
And it is not a ranking lever. There is no documented mechanism by which removing pages from the index raises the ones that remain. The case for a smaller index is a crawl-efficiency case on a large site, and on a site of a few hundred pages there is no case at all.
Stop when the Page Indexing report says the rule was seen. That is the whole outcome. There is no second state where the page becomes more removed.
Common questions
What does noindex mean?
It is a rule, set as a meta tag or an HTTP header, that tells a search engine not to include a page in its results. Google describes the effect as dropping the page "entirely from Google Search results, regardless of whether other sites link to it" — but only once the crawler has fetched the page and read the rule.
noindex or disallow?
noindex if you care whether the page is listed. Disallow if you care whether it is fetched. Using both means the crawler never reads the noindex, which is the single most common way this goes wrong. We measured how rarely either signal is used on homepages: 7 of 28 ship a meta robots tag, and 1 of 27 sends an X-Robots-Tag.
Does noindex remove a page from other engines?
Only from engines that support the rule and have recrawled the page. Google's own wording admits the limit. Treat a noindex as a request understood by most crawlers rather than a switch.
Should I noindex tag and category pages?
Usually not. They are how a crawler discovers the pages beneath them, and removing them from the index does not remove them from that job — but it does remove any chance they rank for the term they are named after. Fix them or merge them first; noindex is the last option, not the tidy one.
Part of the QueryWin handbook · Level 3
