Does crawl budget really only concern very large sites?

What is crawl budget and why does Google set this threshold at one million URLs?

The crawl budget refers to the number of pages Googlebot is willing to explore on a site during a given time period. This quota depends on two main factors: the server's ability to respond quickly without overloading, and the interest that Google has in the site's content.

Martin Splitt sets the critical threshold at one million URLs. Below this, sites typically have a crawl budget that is more than sufficient for all their strategic pages to be visited regularly. Beyond that, Google's prioritization mechanisms become real constraints — certain sections may be ignored or crawled with too much time in between.

Why are so many SEOs concerned about crawl budget when their site is far from a million pages?

Because the diagnosis is often misplaced. Many attribute to crawl budget issues that are actually due to technical faults: catastrophic server response times, endless redirect chains, facets and parameter URLs artificially inflating the number of pages exposed to crawling.

Google does not block the crawl of your site with 50,000 product listings because it decided to ration its budget. It slows it down because your server responds in 2 seconds, or because you expose 200,000 URLs generated by filters that add no value. The problem is not the quota — it's infrastructure and information architecture.

What are the real indicators to watch for below one million URLs?

Instead of fantasizing about crawl budget, focus on tangible metrics. The crawl rate of strategic pages in Search Console, the frequency of Googlebot visits to your new posts, and server 5xx errors detected during crawling.

If your important pages are crawled every day or several times a week, and your new content is indexed within hours, you do not have a crawl budget issue. If on the contrary, key URLs remain ignored for weeks, look into the internal linking architecture, improperly configured robots.txt files or sitemaps, or weak quality signals that hinder Google's appetite.

• Crawl budget only becomes critical beyond one million indexable URLs
• Below this threshold, crawl slowdowns usually stem from server technical deficiencies or a faulty site architecture
• The real indicators to monitor: crawl rate of strategic pages, indexing time for new content, server errors during crawling
• Optimizing crawl budget on a medium-sized site often involves resolving issues related to server performance, redirect chains, and unnecessary facets
• Prioritize the quality of the Googlebot experience over the quantity of pages crawled

Is this statement consistent with observed practices on the ground?

Overall, yes. Field audits confirm that sites with between 10,000 and 500,000 URLs rarely suffer from crawl budget restrictions in the strict sense. When strategic pages are not crawled, the explanation almost always lies in negative signals: orphan pages with no internal linking, mass duplicated content, sluggish server responses, poorly placed noindex directives.

The hitch is that Martin Splitt does not specify how long Google tolerates these failures before actively throttling the crawl. A server that regularly returns 503 errors or response times > 3 seconds will see its crawl throttled even on a site with 20,000 pages. The nuance matters: Google does not say that crawl budget does not exist below one million, it says it should not be the limiting factor — provided everything else is clean.

In what cases does this one million URLs rule not apply?

First case: sites with an extreme publishing velocity. A news media outlet producing 500 articles per day can reach 180,000 URLs per year, but if Google only crawls every 48 hours, the news loses relevance before indexing. Here, the issue is not so much the total volume but the frequency of crawl — and this, Martin Splitt does not address.

Second case: architectures with multiple poorly managed subdomains or international versions. Google allocates its budget by hostname. If you fragment your 300,000 pages across 15 technical subdomains without SEO logic, each subdomain ends up with a reduced budget — and some sections may be under-crawled even if the total remains below a million.

Third case — and this is where it gets tricky: Google remains vague about the exact definition of this million. Discovered URLs, URLs in the sitemap, indexed URLs, canonical URLs? The answer changes everything. [To be verified] Does a site with 200,000 canonical pages but 2 million faceted URLs exposed to crawling fall into the "very large sites" category?

What nuances should be added to this official position?

Google deliberately simplifies to prevent every WordPress blog webmaster from getting bogged down with crawl budget. But this simplification masks more complex realities. Crawl budget is a result of several factors: site popularity, perceived authority, content freshness, technical health, user signals.

Two sites with 500,000 pages will not receive the same treatment. A reference media site with enormous traffic and solid backlinks will benefit from a more generous crawl than a low-quality directory that has been artificially inflated. Saying that crawl budget is not an issue below one million is true for a technically flawless site with strong authority. For others? The real threshold may be much lower.

What should I do concretely if my site has fewer than one million pages?

First, stop optimizing for a problem that probably doesn't exist. Too many SEOs waste time dissecting server logs to track even the slightest passage of Googlebot, while their real issue lies elsewhere: weak content, keyword cannibalization, shaky silo structure.

Next, invest in what really matters: the technical health of the server. Hosting that can respond in less than 500ms even under load, clean cache management, well-configured CDNs. Google crawls a fast and stable site more generously than a slow site, even if the latter has few pages.

What mistakes should I avoid to not artificially create a crawl problem?

First classic mistake: exposing facets and filters without limits via internal linking or sitemaps. You turn 10,000 product listings into 300,000 combinatorial URLs that Googlebot will try to crawl, diluting its attention. The result: strategic pages are crawled less often, not due to lack of overall budget, but due to poor allocation of this budget.

Second mistake: neglecting the robots.txt file and noindex/nofollow directives. Entire sections of the site can be accidentally blocked, creating the illusion of insufficient crawling when it is you who are closing the door. Conversely, letting Google explore thousands of empty internal search pages or tags without content wastes crawl time for nothing.

How can I check that my site is not suffering from a disguised crawl problem?

Open Search Console, crawl statistics section. Look at the number of crawl requests per day, average response times, the server error rate. If these metrics are stable and your key pages regularly appear in the logs, you’re in the clear.

Next, analyze your server logs — not to obsessively track every bot, but to identify anomalies. Sections ignored for weeks? A crawl focused on worthless URLs? These are symptoms of a failing architecture, not insufficient crawl budget. Fix the linking, clean up the sitemaps, reinforce internal signals towards important pages.

• Audit server performance: response times < 500ms, server error rate close to zero
• Identify and block unnecessary facets generating combinatorial URLs without added value
• Check that your strategic pages are crawled regularly through Search Console and server logs
• Clean up sitemaps to only submit indexable and high-value URLs
• Reinforce internal linking towards priority content to guide crawl allocation
• Monitor indexing delay of new content: if it exceeds 48 hours for important pages, dig into the technical causes