Does robots.txt really prevent the indexing of your pages?

What’s the difference between blocking crawling and blocking indexing?‍

The robots.txt file is an exclusion directive that tells robots not to crawl certain URLs. Blocking crawling means that Googlebot will not download the page, will not analyze its HTML content, and will not explore its internal links.
Indexing, on the other hand, involves adding a URL to Google’s index so it can appear in search results. Google can index a URL without having crawled it — simply because it is mentioned on other sites through backlinks or external references.

How can Google index a page without crawling it?‍

In practice? If your page /admin/dashboard is blocked by robots.txt but 15 external sites link to it with descriptive anchor text, Google will discover this URL. It won't be able to access the content — but it knows of the URL, its anchor text, and can decide to index it.

These pages appear in results with the note "No information available on this page" — no meta description, no snippet, just the URL and sometimes the anchor text from backlinks. You will mainly see them in site: queries during an indexing audit.

Why don’t these pages appear in normal searches?‍

Google states that these URLs blocked by robots.txt but indexed generally do not rank for standard queries. The reason is simple: without analyzed content, there are no relevance signals — no detected keywords, no semantic structure, no on-page analysis.

The engine can hardly assess the thematic relevance of a page whose HTML it has never read. These URLs remain invisible to the average user but may potentially pollute your crawl budget and create noise in the index.

• Robots.txt blocks crawling, not indexing — a fundamental distinction often misunderstood
• Google can index a URL discovered through external backlinks even if it is blocked
• These pages appear without a snippet in site: queries but rarely in regular SERPs
• To actually block indexing, you must use noindex in meta robots or HTTP header
• Paradox: to apply a noindex, Google must be able to crawl the page — so robots.txt and noindex are incompatible

Is this statement consistent with real-world observations?‍

Yes, and it’s even a recurring issue in SEO audits. We regularly observe sites that block entire sections via robots.txt — thinking they will be invisible — and then find these URLs indexed with rogue backlinks or mentions in directories. Mueller’s statement is factual and documented.

Let’s be honest: many beginner SEOs still believe that robots.txt = total protection. This is false. If you want to prevent Google from indexing a sensitive page, robots.txt is the worst possible tool — it actually blocks the access that would allow reading your noindex tag.

What nuances should be added to this statement?‍

Mueller clarifies that these pages "generally do not rank" — which leaves room for interpretation. In reality, we see that some blocked URLs with strong backlinks can appear for very specific brand or navigational queries. [To be verified] The actual impact on organic traffic remains marginal in 99% of cases.

Another nuance: Google talks about "normal queries," but does not precisely define what a normal query is versus a site: query. In practice, this means that the indexing audit via site: will reveal these URLs, but they do not actively pollute commercial SERPs. This remains problematic for the clarity of your index.

In what cases does this rule cause tangible problems?‍

Classic case: you migrate a site and block the old domain via robots.txt while waiting for the redirects. Result? The old URLs remain indexed, Google cannot crawl to discover the 301s, and you create a technical debt hell. The deindexing drags on for months.

Another common scenario: sections like /admin/ or /test/ blocked by robots.txt but linked from a footer or a forgotten XML sitemap. These pages accumulate in the index with empty snippets, Google crawls them in loops (and hits a 403), and you waste crawl budget for nothing.

What should you do to truly block the indexing of a page?‍

The only reliable method to prevent indexing is the meta name="robots" content="noindex" tag in the of the HTML, or the HTTP header X-Robots-Tag: noindex for non-HTML files (PDFs, images). These directives explicitly tell Google not to index the resource.

Problem: to read this directive, Google must crawl the page. So if you block the page via robots.txt, Googlebot will never discover your noindex — and the page may remain indexed through external links. This is a technical paradox that needs to be anticipated.

How to clean up undesirable URLs already indexed via robots.txt?‍

First step: temporarily remove the robots.txt block to allow Google to crawl these pages and discover the noindex tags you will add. Yes, it sounds counterintuitive — but it’s the only way to clearly communicate your intent to the engine.

Once the pages are crawled and the noindex detected, Google will gradually remove them from the index. You can speed up the process through Search Console by requesting a temporary removal (effective within 24 hours), then let the noindex handle the permanent removal. Finally, you can restore the robots.txt block if necessary — but at this point, the noindex will have done its job.

What critical mistakes should you avoid in managing crawling and indexing?‍

Critical mistake #1: blocking by robots.txt AND adding noindex. The two directives are incompatible — robots.txt prevents reading the noindex. Choose one or the other based on your goal: block crawling (saving budget) or block indexing (removing URLs from SERPs).

Critical mistake #2: using robots.txt as a protection for sensitive content. If you have confidential data, the solution is HTTP authentication (login/password) or server protection — never robots.txt, which is a simple recommendation that any malicious robot can ignore. Google respects robots.txt, but not scrapers or unscrupulous competitors.

• Regularly audit site: queries to detect indexed URLs without a snippet — a sign of a robots.txt block
• Identify backlinks or external references to blocked pages (Search Console, Ahrefs, Screaming Frog)
• Temporarily remove the robots.txt block to allow crawling and application of noindex tags
• Use Search Console to force a temporary removal of unwanted URLs while the noindex propagates
• Never block entire sections via robots.txt if they contain 301/302 redirects that Google needs to discover
• Document your blocking choices in an SEO management file — too many sites accumulate obsolete and contradictory robots.txt rules