Key takeaways
- noindex removes a page from search results. It needs Google to crawl the page to see the instruction.
- Disallow in robots.txt blocks crawling, not indexing. A blocked URL can still appear in Google if other pages link to it.
- The deadly combo is noindex plus disallow together. The disallow rule stops Google from ever crawling the page, so it never sees the noindex, and the URL can stay in the index indefinitely.
- To reliably keep a page out of Google, let it be crawled and apply noindex, password-protect it, or remove it. Then confirm in Search Console.
Two of the most common tools for controlling what Google shows are also the two most commonly confused. noindex and robots.txt disallow sound like they do the same job, so people reach for whichever comes to mind, and often use both at once for good measure. That instinct is exactly what backfires. They operate at different stages of how search engines work, and using them together can permanently lock an unwanted page into search results rather than removing it.
This guide explains what each one actually does, why combining them is a trap, how to genuinely remove a page, and how to decide which tool fits the job. If you want the wider context on crawling, indexing and rendering, start with our complete technical SEO guide.
What noindex does (and what it doesn’t)
noindex is a rule that tells search engines to keep a page out of their results. You can apply it two ways, and both have the same effect: as a meta tag in the page’s HTML, or as an HTTP response header. According to Google’s documentation on blocking search indexing with noindex, when Googlebot crawls the page and reads the rule, Google drops that page entirely from Search, regardless of whether other sites link to it.
The meta tag version lives in the head of the page:
<meta name="robots" content="noindex">The header version is useful for non-HTML files such as PDFs, images and videos, where you cannot add a meta tag:
X-Robots-Tag: noindexHere is the part that matters most, and that the same Google documentation states plainly: for the noindex rule to work, the page must not be blocked by a robots.txt file and must otherwise be accessible to the crawler. If the crawler cannot reach the page, it never sees the noindex, and the page can still appear in search results. noindex is an instruction Google can only follow if it is allowed to read it.
What robots.txt disallow does (and what it doesn’t)
A robots.txt file tells crawlers which URLs they may access on your site. Per Google’s robots.txt introduction, it is used mainly to avoid overloading your site with requests. A basic disallow rule looks like this:
User-agent: *
Disallow: /private/The crucial distinction, in Google’s own words, is that robots.txt “is not a mechanism for keeping a web page out of Google.” Disallow controls crawling, not indexing. Google won’t crawl the blocked content, but it might still find and index a disallowed URL if that URL is linked from other places on the web. When that happens, the URL address, and potentially anchor text from links pointing to it, can still show up in search results, usually without a proper title or description.
This is the well-known “Indexed, though blocked by robots.txt” status you may have seen in the Search Console Page indexing report. It is Google telling you it indexed a URL it was never allowed to crawl. One more note worth flagging: Google stopped supporting the unofficial noindex directive inside robots.txt files as of September 1, 2019, as Search Engine Land reported. If you ever relied on that, it no longer works.
The trap: why noindex plus disallow keeps the page in Google
Common mistake. Here is the combination that causes the most damage. You want a page gone, so you add a noindex tag to it. To be extra safe, you also disallow it in robots.txt. It feels like a belt-and-braces approach. It is the opposite.
The disallow rule tells Google not to crawl the page. The noindex rule lives inside the page. So Google, obeying the disallow, never fetches the page, which means it never reads the noindex sitting inside it. As Search Engine Journal summarized in its December 2024 write-up of Google’s guidance on when to use noindex versus disallow: by combining disallow and noindex, you tell Google not to visit the page, and because it can’t visit the page, it never sees your instruction not to index it.
The result is the worst of both worlds. The noindex you were counting on is invisible. And because a disallowed URL can still be indexed from external links, the page can sit in Google’s index indefinitely, with no clean way for Google to learn it should come out. This exact scenario turns up regularly in Google’s own help forums, where owners report a page still appearing in Search despite having applied both a disallow rule and a noindex tag, as in this Google Search Central Community thread. The fix is always the same: unblock the page so Google can crawl it and finally see the noindex.
How to actually remove a page from Google
Google’s robots.txt guide lists the methods that genuinely keep a URL out of results: block indexing with noindex, password-protect the page, or remove it entirely. Pick based on what you actually need.
- Make sure the page is crawlable. If it is disallowed in robots.txt, remove that disallow rule. Google needs to fetch the page to act on anything inside it.
- Apply noindex. Add the
<meta name="robots" content="noindex">tag, or return anX-Robots-Tag: noindexheader for non-HTML files. Leave it in place. - Wait for a recrawl, or request one. The page drops out once Google recrawls it and reads the rule. You can use URL Inspection in Search Console to request indexing of the updated page.
- Need it gone fast? Use the Removals tool in Search Console. Per Google’s Removals tool documentation, it hides a URL from results for about six months, which buys you time to put a permanent method (noindex, password protection, or deletion) in place before the block expires.
- For content that must never be public, password-protect it or remove it from the server entirely. That is the only approach that keeps a page out of the index no matter who links to it.
The recurring lesson: the Removals tool and noindex both depend on Google being able to reach the page. As Google notes, a temporary removal does not stop crawling; it only hides the URL from results, so you still need a durable method underneath it.
When to use which
Match the tool to the goal. The short version: reach for noindex when a page can be crawled but should not appear in results, and reach for disallow when you want to save crawl resources on pages you don’t care about indexing.
| Your goal | Use | Why |
|---|---|---|
| Keep a page out of search results | noindex (page stays crawlable) | Google must crawl the page to see and honor the rule |
| Stop crawling to save server load or crawl budget | robots.txt disallow | Blocks the fetch, but does not guarantee the URL stays out of the index |
| Remove a page permanently and privately | Password protection or full removal | Works regardless of external links; nothing to crawl or index |
| Hide a URL urgently while you fix the real cause | Search Console Removals tool | Temporary, roughly six months; needs a permanent method behind it |
| Keep a page both crawlable and out of results | noindex only, never noindex plus disallow | A disallow rule would hide the noindex from Google |
Two rules cover almost every case. First, never disallow a page you are trying to noindex. Second, if a page is already indexed and you want it out, the disallow has to come off before Google can process the noindex. For how these choices fit alongside canonical tags, sitemaps and the rest of your crawl setup, see our technical SEO guide.
Frequently asked questions
Does robots.txt disallow remove a page from Google?
No. Disallow blocks crawling, not indexing. Google states that a disallowed URL can still be found and indexed if it is linked from elsewhere, so the URL may keep appearing in results. To remove a page, use noindex, password protection, or removal.
Why is my page still indexed after I added noindex?
The most common reason is that the same page is also disallowed in robots.txt. Google cannot crawl it, so it never reads the noindex. Remove the disallow rule, let Google recrawl, and the noindex will take effect. A recrawl can take time, so request indexing via URL Inspection to speed it up. Also, a page that is heavily linked from 3rd party sites might still be indexed even though it has a Noindex tag.
Should I ever use noindex and disallow on the same page?
No. It is the combination to avoid. The disallow prevents Google from seeing the noindex, which defeats the purpose and can leave the URL stuck in the index. Choose one based on your goal: noindex to keep a crawlable page out of results, disallow to prevent crawling of pages you don’t need indexed.
What is the fastest way to hide a page from Google?
The Search Console Removals tool hides a URL from results in hours, but only for about six months, according to Google. Treat it as a stopgap. Pair it with a permanent method such as noindex or password protection so the page does not reappear when the temporary block expires.
Can I put a noindex rule in robots.txt?
Not anymore. Google stopped supporting the unofficial noindex directive in robots.txt on September 1, 2019, as Search Engine Land reported. Use a meta robots tag or an X-Robots-Tag HTTP header instead.
Written by Matthis Duarte, a senior SEO and organic growth expert with 10+ years of experience driving organic growth for international brands across highly competitive verticals. He is the founder of Knownful, an independent publication on SEO and organic growth featuring in-depth guides, best practices, playbooks and original analyses, including a free monthly study of which brands AI actually recommends across 10 industries.
