Why doesn't robots.txt deindex your pages, and which method should you choose to remove URLs from the index?

What’s the difference between blocking crawling and preventing indexing?

The confusion arises from a common technical misunderstanding. Blocking a URL in robots.txt prevents Googlebot from visiting the page and reading its content. But if there are external backlinks pointing to this URL, Google can still index it based solely on those external signals: anchor text, link context, popularity.

The result: you'll find in the index pages with a generic snippet like 'No information available' or just the displayed URL. Robots.txt protects the content but not the presence in the index. This is particularly evident on sensitive pages blocked late after accumulating links.

Why does Google index what it can't crawl?

Google builds its index from multiple signals beyond the page content. An external link is a signal of existence: if 50 sites mention a URL, Google considers it worthy of appearing in the results, even without having read its content.

This logic aims to prevent relevant contents from disappearing from the index due to server configuration errors or poorly configured robots.txt files. But it creates a trap: blocking a page in robots.txt that has already been indexed does not remove it. Worse, you prevent Google from seeing a potential noindex tag you may have added.

What are the two methods recommended by Google?

The removal tool in Search Console acts within hours to temporarily remove a URL from the index. Effective for emergencies (duplicate content in production, data leaks), but limited in duration: the effect lasts 6 months, after which Google may reindex if the page remains accessible.

The noindex tag in HTML or in HTTP headers is the long-term solution. It explicitly tells Google not to index the page, even if external links point to it. Unlike robots.txt, it requires Googlebot to access the page to read the instruction. Once applied, deindexing occurs at the next crawl and continues as long as the directive remains in place.

• Robots.txt blocks crawling, not indexing: ineffective for removing a page from the index
• Search Console removal tool: quick but temporary solution (6 months), ideal for emergencies
• Noindex tag: definitive and natural method, requires the page to remain crawlable
• A page blocked in robots.txt can remain indexed if external backlinks reference it
• Adding a noindex to a page blocked in robots.txt is pointless: Googlebot will not be able to read it

Is this distinction between crawling and indexing always respected by Google?

In principle, yes. Field observations confirm that a URL blocked in robots.txt but widely linked often still appears in the index with an empty snippet. But the reality is more nuanced. If a page has never been crawled before the block and receives only very few low-quality links, it may never get indexed.

Conversely, pages with a crawling history and a strong link profile persist in the index despite recent robots.txt blocking. The timeline for natural deindexing varies greatly: from a few weeks to several months depending on the page authority. Google does not instantly deindex what it can no longer crawl. [To be verified] in what exact proportion this persistence occurs across different site types.

Is noindex always the best long-term solution?

In 95% of cases, yes. Noindex remains the cleanest and most explicit directive. However, there are situations where this approach poses challenges. If you need to deindex thousands of dynamically generated pages (facets, filters, unnecessary pagination), adding a noindex to each one consumes crawl budget unnecessarily.

In such cases, a combination might be more effective: noindex on high crawl potential pages, robots.txt on entire low-value sections. But be careful: if you block in robots.txt after indexing, deindexing will only occur gradually, through the natural expiration of Google's cached data. This process is slow and unpredictable.

Does the Search Console removal tool pose risks?

Yes, and this is rarely mentioned. A hasty removal without a sustainable solution behind it creates a yo-yo effect. If you remove a URL via the tool but do not implement a noindex, Google will reindex it as soon as the 6-month deadline expires, and you'll have to repeat the operation manually.

Worse: on some sites with high content turnover, the removal tool can generate a history of requests that is difficult to track. If you remove 200 URLs per month, you need to monitor their reindexing 6 months later. In practice, the tool is excellent for occasional emergencies but becomes unmanageable at scale without automation or a rigorous process. [To be verified] if Google plans to extend the removal duration or automate renewals.

How to audit and correct blocked URLs still present in the index?

First step: identify the pages blocked in robots.txt but indexed. Use the query site:yourdomain.com in Google and manually filter the URLs that shouldn’t appear or cross-reference your sitemap with the Search Console data (Coverage tab). The pages marked 'Excluded by robots.txt' but present in the index are your top targets.

Next, decide for each URL: was it really necessary to block it? If so, remove the corresponding line from robots.txt and add a noindex tag in HTML or in HTTP header. Wait for Google to crawl the page again to read the noindex. You can speed this up by requesting a reindexing via the URL inspection tool in Search Console. Once the noindex is detected, the page will naturally disappear from the index on the next refresh.

What strategy to adopt for emergency removals vs. planned cleanses?

In emergencies (data leaks, defamatory content, massive duplication), combine both tools: initiate a removal via Search Console for immediate effect, then add noindex to maintain. You gain responsiveness while laying the groundwork for a sustainable solution.

For a planned structural cleanup (revamps, removal of obsolete categories, elimination of thin content), prioritize noindex only. Plan the rollout in waves to monitor the impact on organic traffic and crawl budget. Follow the progressive deindexing in Search Console: some pages will leave in a few days, others in several weeks depending on their crawl frequency.

What common mistakes should absolutely be avoided?

The most frequent mistake: adding a noindex then blocking the page in robots.txt. You create a conflict: Google can no longer access the page to read the noindex, so the directive becomes useless. If the page had backlinks, it remains indexed indefinitely with an empty snippet. Always check that pages with noindex remain crawlable.

The second trap: using the Search Console removal tool as a definitive solution. Six months later, the URLs reappear and you no longer receive notifications. Document each removal and schedule a reminder to check the page's status before expiration. Or better yet: consistently deploy a noindex in parallel to prevent automatic reindexing.

• Audit pages blocked in robots.txt but still indexed via site: and Search Console
• Remove from robots.txt and add an HTML or HTTP noindex for sustainable deindexing
• Use the Search Console removal tool only for emergencies, never as a definitive solution
• Never block in robots.txt a page with a noindex: the bot won't be able to read the directive
• Monitor progressive deindexing in Search Console after adding the noindex
• Document each temporary removal to anticipate automatic reindexing after 6 months