Does the noindex tag really block the crawling of your pages?

Why does Google need to crawl a noindex page? 

The logic is simple: Google cannot know about a noindex directive without accessing the page. The HTTP X-Robots-Tag header is sent by the server in response to a request, not before.

Unlike the robots.txt file that gives instructions before any crawl, noindex is an indexing directive discovered during the crawl. The bot must visit the page, receive the header, analyze the directive, and then decide not to index the content. It is only after several visits that Googlebot understands the pattern.

What actually happens when implementing a noindex? 

Mueller clarifies that crawlers will first heavily visit these pages. It is a learning phase for Google's algorithms, which must identify that these URLs lead to "nowhere" in terms of indexing.

Gradually, the systems adjust their priorities. Crawling naturally slows down, as Google optimizes its crawl budget by reducing the frequency of visits to pages it knows are non-indexable. However, this reduction is never total—Google will periodically revisit to check if the directive is still present.

How does this differ from robots.txt or the meta robots? 

The robots.txt file blocks crawling upfront, even before the bot loads the page. If a URL is disallowed, Googlebot does not visit it (unless it receives backlinks, in which case it can be indexed without content). This is a mechanism for preventive filtering.

The meta robots tag in the HTML works like X-Robots-Tag: Google must crawl the page to read it. The difference? X-Robots-Tag is an HTTP header, useful for non-HTML files (PDFs, images) or for directives applied at the server level without modifying the source code.

• Noindex never replaces robots.txt to save crawl budget from the start
• Google gradually learns which pages are marked noindex and adjusts its behavior
• Noindex pages continue to be crawled sporadically to check the persistence of the directive
• The X-Robots-Tag header is discovered during crawling, never before
• Combining robots.txt and noindex is redundant and counterproductive if the goal is to deindex a URL already indexed

Is this statement consistent with real-world observations? 

Yes, absolutely. It is regularly observed in logs that noindex-marked pages are heavily crawled in the first few days following their implementation. Googlebot tests, returns, retests—it's a classic pattern.

What is interesting is the official confirmation of the gradual learning mechanism. On sites with several hundred thousand pages, we have seen periods of 3 to 6 weeks before the crawling of noindex sections stabilizes at a low level. Patience is therefore required—the crawl budget does not free up overnight.

What nuances should be considered regarding this rule? 

The first point: not all noindex pages are equal. A page receiving backlinks or linked from important areas of the site will be crawled more often, even with a noindex. Google returns to check if the directive is still there, especially if the page has "weight" in the link graph.

The second point: the reduction of crawling is never absolute. Mueller says "they gradually reduce crawling," not "they stop crawling." Googlebot will always return, if only to ensure that the directive has not changed. On an active site, expect a visit every 15 to 45 days depending on the URL's popularity. [To be verified]: Google does not publish precise data on residual frequency.

In what situations does this logic cause problems? 

The classic case: you have 50,000 poorly managed faceted filter pages that are already indexed. You add a noindex via X-Robots-Tag. Result? Google will massively crawl these 50,000 pages to discover the noindex, which can saturate your crawl budget for several weeks.

If these pages were blocked in robots.txt, they would remain indexed (Google cannot crawl to see the noindex), but at least the crawl budget would be preserved. Let’s be honest: rapid deindexing has a cost in crawl. Sometimes you have to choose between the speed of cleaning the index and preserving server resources. On sites with a tight crawl budget, this learning phase can delay the crawling of strategic pages.

What concrete steps should be taken to manage noindex pages? 

Accept the initial intense crawl phase. If you implement a noindex on an entire section, monitor your server logs and your Search Console. You will see a spike in crawling in the following days—that's normal and expected.

Prioritize your efforts: if you have a limited crawl budget (large site, low authority), introduce noindex in batches rather than all at once. For example, mark 10,000 pages per week rather than 100,000 all at once. This smooths the impact on crawling and avoids saturating your server resources.

What mistakes should be avoided when using noindex? 

Never combine robots.txt disallow and noindex on the same URL if the goal is to deindex a page already in the index. Robots.txt prevents Google from seeing the noindex, so the page remains indexed as a content-less URL.

Another trap: removing the noindex too soon. If you mark a section as noindex and then change your mind 10 days later, you restart the learning cycle. Google must recrawl, understand that the noindex has disappeared, reevaluate indexability. Be sure of your decision before deploying.

How can you check the effectiveness of noindex on your crawl budget? 

Use the crawl stats reports in Search Console. Filter by response type to track pages returning a noindex. You should see a curve: a rapid increase in the number of crawls on these URLs, followed by a gradual decrease over 4 to 8 weeks.

Analyze your server logs to confirm that Googlebot is indeed reducing its visits. If after 2 months you still observe sustained crawling, check that these pages are not receiving external backlinks or are not linked from hot areas of the site. Internal linking influences the persistence of crawling even with noindex.

• Monitor the initial crawl spike in the logs and Search Console after deploying noindex
• Deploy in phases if you have thousands of pages to mark as noindex to smooth the impact
• Never block in robots.txt a page you want to deindex with noindex
• Wait 6 to 8 weeks before judging the effect of noindex on the crawl budget
• Check the internal linking to noindex pages to limit conflicting signals
• Document your decisions to avoid back-and-forths that reset Google’s learning