Why does Google only crawl a fraction of your known pages?

Does Google really crawl all the URLs it knows?

No, and it never has. Google only crawls a fraction of known URLs from a site, regardless of its size. This reality is often misunderstood: submitting 100,000 pages via sitemap does not guarantee that these pages will be visited by Googlebot.

The search engine performs an active selection based on its perception of the site's quality and the relevance of each URL. If Google determines that 80% of your pages are not valuable, it will not waste time crawling them regularly — or even at all.

What determines the allocated crawl volume?

The crawl budget is not a fixed quota: it’s a dynamic allocation that reflects the trust Google places in your site. The higher the perceived quality, the more resources Googlebot dedicates to exploring your content.

Specifically? A site with unique content, regularly updated, and technically sound will see its crawl volume gradually increase. Conversely, a site filled with duplicate pages, low-quality content, or unnecessary facets will see its budget stagnate — or even regress.

Why has this limitation always existed?

Because crawling the web is costly in server resources, bandwidth, and energy. Google has to prioritize: it cannot visit every page of every site on the web daily, especially when 90% of crawled content is not worthy of being indexed.

This economic constraint forces Google to be selective from the crawl stage. It’s a barrier even before indexing: if a page is never crawled, it cannot compete for SERP rankings. And this is where many SEOs go wrong: they optimize pages that Google simply does not visit.

• Google only crawls a portion of known URLs, even via XML sitemap
• This crawl volume is proportional to the perceived quality of the site
• An uncrawled URL cannot be indexed, regardless of its intrinsic qualities
• This limitation has existed since the inception of Google and is not a recent phenomenon
• Improving the overall quality of the site mechanically increases the allocated crawl budget

Is this statement consistent with real-world observations?

Yes, and it’s even one of the few statements from Google that perfectly aligns with actual SEO audits. On sites with 50,000+ pages, it’s common to see 40% to 60% of the URLs never visited by Googlebot, even after several months online.

The problem is that many SEOs discover this reality too late — after generating thousands of low-value filter or category pages. They then see that Google completely ignores these URLs, without even crawling them once.

Why is Google vague about the exact thresholds?

Because there is no universal rule. The crawled volume depends on dozens of factors: domain history, content popularity, update frequency, technical quality, page depth, server speed, HTTP error rates...

Google does not want to provide precise numbers to prevent SEOs from trying to game the system. But concretely? A typical e-commerce site with 200,000 products will rarely have more than 30% to 50% of its pages crawled regularly. [To be verified] on your own project via server logs.

What are the limits of this overall quality logic?

The issue is that Google judges quality at the site-wide level, not page by page at the initial crawl. If 80% of your site is mediocre, even your 20% premium pages might never be crawled simply because they are drowned in the mass.

This is where a mass cleanup strategy comes into play: deindexing or removing weak pages can paradoxically improve the crawl of important pages. Some sites have doubled their organic traffic by removing 60% of their content — this is not a myth, it’s a real-world reality for large sites.

How to effectively measure your site's crawl budget?

The first step is to analyze your server logs. Google Search Console provides a partial view, but raw logs show you exactly which URLs are visited, how often, and with what depth.

Then cross-reference this data with your declared XML sitemap. If you have 50,000 submitted URLs but only 10,000 crawled over 30 days, you have a structural issue. Either your content is deemed weak, or your architecture is drowning important pages.

What concrete actions can increase crawled volume?

First priority: eliminate low-value pages. Unnecessary facets, duplicate pages, thin content, empty categories — everything that pollutes the crawl without driving traffic should be deindexed or removed.

Next, optimize your internal linking to push strategic pages: an orphan page or one located 8 clicks from the homepage is unlikely to be crawled regularly. Bring your key content within 2-3 clicks maximum through relevant contextual links.

Finally, improve your technical signals: server speed, response time, 4xx/5xx error rates, unnecessary redirects. A slow or unstable server mechanically lowers your crawl budget — Google does not want to overload your resources.

What critical mistakes should you absolutely avoid?

Number one mistake: massively generating pages without ensuring they will be crawled. Before launching 100,000 product sheets or 500,000 filter combinations, verify that your site has the technical and qualitative capacity to handle this volume.

Number two mistake: ignoring signs of excessive crawling. If Google crawls 80% of your pages but only 20% generate traffic, you’re wasting budget on unnecessary content. Redirect this budget towards your strategic pages by cleaning up the rest.

• Analyze your server logs to identify the actual crawl rate versus known URLs
• Remove or deindex all low-value pages (thin content, duplications, unnecessary facets)
• Optimize your internal linking to elevate strategic pages within 2-3 clicks of the homepage
• Enhance server speed and response time to maximize crawl efficiency
• Only submit your best pages in the XML sitemap — not the entire site
• Monitor the evolution of the crawl budget via Search Console and logs after each optimization