Why is robots.txt not enough to block the indexing of your pages?

What is the difference between crawling and indexing?

Crawling refers to the process where Google's bot explores a page to extract its content. Indexing is the decision to include that page in the searchable database of the search engine.

Robots.txt blocks crawling—the bot cannot access the page. But if there are external links pointing to this URL, Google can still index it with the available information (anchor text, link context). As a result, a URL may appear in the SERPs with an empty or generic snippet, even though Google never read the content.

Why does this confusion persist among so many practitioners?

Because for years, blocking crawling via robots.txt used to work indirectly for certain pages. If Google didn't crawl, it often didn't index either—but this was never guaranteed.

The problem arises when third-party backlinks signal the existence of a blocked URL. Google then creates a minimal index entry, based solely on external signals. Hence, you find your private URL in the results, with a rough title taken from the anchor text.

In what concrete cases does this situation occur?

Typically on staging environments blocked by robots.txt but linked from an external site, or admin pages erroneously referenced in forums or third-party tools.

Misconfigured CMSs also generate technical URLs blocked from crawling but indexed through contradictory XML sitemaps. Google sees the URL in the sitemap, notes that it is blocked from crawling, but still indexes it if other signals deem it relevant.

• Robots.txt blocks crawling, not indexing—a URL can appear in the SERPs without being crawled
• Noindex is the technical directive to exclude a page from the index (requires Google to crawl the page to read the tag)
• Authentication (login/password) physically prevents access—an extreme but effective method
• If a URL is blocked by robots.txt AND has backlinks, Google can create a partial index entry based on anchor texts
• The combination of robots.txt + noindex is technically contradictory—Google cannot read the noindex if it does not crawl

Is this statement consistent with real-world observations?

Yes, and it's even a welcome reminder. In practice, we regularly see URLs blocked by robots.txt that still appear as indexed in Google Search Console. The mention "Blocked by robots.txt" in GSC specifically refers to these ambiguous cases.

What is missing in Waisberg's statement is an explanation of the deindexing delay. Switching from robots.txt to noindex on an already indexed page does not guarantee immediate removal from the index—Google must first recrawl to read the noindex. [To be verified] : no official data on the average duration of this process.

What concrete risks exist for an e-commerce or editorial site?

The classic scenario: a site blocks its filter facets or internal search results pages via robots.txt to preserve crawl budget. If these URLs receive external links (from forums, comparison sites), they can get indexed with empty or misleading snippets.

Result: unintentional cannibalization and dilution of visibility. Google presents a useless technical URL instead of the strategic category page. Worse, these ghost URLs consume crawl budget during periodic recrawl attempts, even if they remain blocked.

In what cases does the rule not strictly apply?

If a URL has no external backlinks and is not listed in any XML sitemaps, blocking it via robots.txt usually suffices to avoid indexing. But it's a gamble—you have no contractual guarantee from Google on this point.

Another exception: purely technical resources (CSS, JS) that you block for performance reasons. Google recommends not to block these resources, but if you do, they are unlikely to appear as organic results anyway. The problem remains limited to HTML pages intended for users.

What concrete actions should be taken on an existing site?

First step: cross audit robots.txt / Google index. Use the command site:yourdomain.com in Google, then filter for URLs that should be blocked. Compare with your robots.txt file—any blocked URL that still appears in the SERPs needs a noindex.

Next, check in Google Search Console the section "Pages" > "Why are the pages not indexed". Specifically look for the label "Blocked by robots.txt". If these pages have impressions or clicks, it means they are paradoxically indexed despite the blocking.

What mistakes to avoid when migrating to noindex?

Never remove a robots.txt line without first adding the noindex and verifying the recrawl. Otherwise, Google will massively crawl the newly accessible URLs and potentially index unwanted content before you can react.

Avoid noindex via X-Robots-Tag HTTP on pages blocked by robots.txt—same issue as the meta tag. Google must be able to access the HTTP response to read the header. The only viable exception remains server authentication (401/403) which physically blocks access.

How to verify that the configuration is correct in the long term?

Set up a Search Console alert for newly indexed pages with the label "Blocked by robots.txt". This detects inconsistencies as soon as they occur, especially after CMS updates or migrations.

Regularly test with the URL inspection tool in GSC: it indicates if a page is blocked from crawling but present in the index. For sites with thousands of technical URLs, automate this check via the Search Console API and monitoring scripts.

• Audit blocked URLs by robots.txt that still appear in site:
• Replace robots.txt with noindex on all pages to be excluded from the index
• Temporarily remove the robots.txt blocking to allow the noindex to be crawled
• Monitor GSC to detect new "Blocked by robots.txt" pages that are indexed
• Prioritize server authentication for truly sensitive content (admin, staging)
• Never combine robots.txt + noindex on the same URL (technical contradiction)