Do URLs blocked by robots.txt really consume your crawl budget?

Why does Google say that blocked URLs do not weigh on crawling?

The principle is simple: Google cannot crawl what it does not have the right to see. When a Disallow directive is present in your robots.txt, Googlebot stops dead. It does not download the page, analyze it, or follow its internal links. The URL is seen, noted, but never actively crawled.

Technically, the crawl budget represents the number of pages that Google agrees to crawl on your site within a given timeframe. This budget depends on the technical health of the site, its popularity, and how frequently the content is updated. If a URL is blocked, it never enters this crawling queue — it simply does not exist for Googlebot as a resource to analyze.

At what point does this question become relevant?

Mueller talks about “millions of URLs”. It's vague, but it gives a sense of scale. For a typical site of a few thousand pages, worrying about the crawl budget is premature. Google can crawl a well-structured site without issues, even with tens of thousands of pages if they provide value.

The real problem arises when you generate useless or duplicate URLs on a large scale: product filter facets, session parameters, infinite paginated pages. In this case, blocking via robots.txt may seem tempting — but it’s not necessarily the best solution. You are hiding the symptom without addressing the cause.

What really consumes the crawl budget then?

Everything Google actually crawls. Slow pages, accessible duplicate content, redirect chains, 404 errors discovered through internal links. Every HTTP request counts. If your site responds slowly or generates a lot of errors, Google slows its crawl to avoid overloading your servers.

URLs blocked by robots.txt do not fall into this category. They do not generate a complete HTTP request, no HTML analysis, no JavaScript rendering. They stay in limbo — visible in some reports but never crawled.

• Blocked URLs by robots.txt do not consume crawl budget because Google never actively crawls them.
• This is only an issue for sites managing millions of URLs, not for the majority of web projects.
• The real waste of crawls comes from slow, duplicate, or inaccessible pages that Google still crawls.
• Blocking via robots.txt hides the problem without solving it — it's better to clean up at the source.
• Focus on the quality and structure of accessible pages rather than on marginal optimizations.

Is this statement consistent with what we observe in the field?

Yes, broadly speaking. Log audits show that Googlebot strictly adheres to robots.txt. Blocked URLs do not appear in crawl logs — or only in HEAD requests to check status, not in complete GET requests. This is consistent with Mueller's statement.

But — and this is where it gets complicated — Google can still index a URL blocked by robots.txt if it discovers it via external backlinks. You will then see it in the Search Console with the note “Indexed, but blocked by robots.txt”. It does not consume crawl, sure, but it pollutes your search results with an empty snippet. That’s not exactly the goal.

What nuances should be added to this recommendation?

Mueller says “don't worry unless you have millions of URLs.” Let's be honest: this wording is vague. From what exact threshold? 1 million? 5 million? And what do we mean by “URL” — those existing or those potentially generated by parameters? [To be verified]

Second nuance: blocking by robots.txt is never the ideal solution. If you're generating millions of useless URLs, the real question is: why do they exist? Poor pagination, uncontrolled filters, session IDs in URLs. Robots.txt becomes a band-aid on a wooden leg. It’s better to fix the architecture or use noindex meta tags (which do require a crawl, sure, but allow for more precise control).

In which cases does this rule not fully apply?

High-velocity sites — news sites, e-commerce with constant refresh — can saturate their crawl budget even with seemingly reasonable volumes. If your site publishes 10,000 new products a day and removes as many, Google may struggle to keep up.

Another edge case: sites with significant server performance problems. Even if blocked URLs don't consume crawl, poor configuration of robots.txt (heavy file, poorly cached) can slow down Googlebot even before it starts to crawl. This is rare, but it can happen on poorly optimized infrastructures.

What practical steps should you take if you manage a large site with millions of URLs?

First step: identify which URLs are truly unnecessary. Analyze your crawl logs, spot sections that consume budget without providing value (sorting pages, redundant filters, obsolete archives). Do not blindly block via robots.txt — first ask yourself why these URLs exist.

Next, prioritize your actions. If these URLs are technically necessary (user filters for instance), but without SEO interest, use a combination of canonical tags to consolidate the signal and meta noindex to avoid indexing. Robots.txt should only come into play as a last resort, for entire sections clearly outside the SEO scope.

What mistakes should you absolutely avoid in managing robots.txt?

Never block critical resources (CSS, JS, images) via robots.txt thinking you are saving crawl. Google needs these to render your pages correctly. Blocking these resources harms the evaluation of your Core Web Vitals and can completely break the indexing of JavaScript-heavy content.

Another classic mistake: blocking already indexed pages. If a URL is in Google's index and you block it via robots.txt, it remains indexed but becomes a “ghost” — present in the SERPs with an empty snippet. To remove it properly, keep it accessible until Google crawls a noindex tag, then block if necessary.

How can you check that your configuration is not negatively impacting crawling?

Use the Search Console: the “Crawl Settings” section (if available in your version) or analyze the coverage reports. Look for URLs “Indexed, but blocked by robots.txt” — this is a warning signal. These URLs pollute your index without you being able to control their presentation.

Compare your server logs with Search Console data. If you see Googlebot attempting to access URLs you thought were blocked, it means your robots.txt is not being interpreted as you wish. Check the syntax, wildcards, and test with Google's robots.txt testing tool.

• Analyze your logs to identify URLs that consume crawl without added value.
• Prioritize structural solutions (canonical, noindex) before blocking via robots.txt.
• Never block critical CSS, JS, or images for rendering.
• Monitor URLs “Indexed, but blocked by robots.txt” in Search Console.
• Test your robots.txt with the official Google tool before any major changes.
• Document each blocking directive to facilitate future audits.