What does your "discovered but not crawled" URL status really reveal about your site?

What exactly does this "discovered not crawled" status mean?

Google has identified the URL — via an internal link, external link, or sitemap — but decided not to crawl it. It's not an oversight: it's a deliberate choice by the algorithm.

This status indicates that Googlebot has prioritized other pages on your site. It estimates that these URLs don't deserve immediate crawling, or perhaps never will.

Why does Google refuse to crawl certain pages?

Two main scenarios according to Gary Illyes: quality problem or technical problem.

If it's a quality issue, Google thinks the content interests no one — duplicate pages, thin content, unnecessary facets, out-of-stock product sheets. If it's technical, your server responds too slowly, timeouts, sporadic 5xx errors. In both cases, Googlebot conserves its crawl budget.

How do you distinguish a quality problem from a technical problem?

Analyze your server logs. If Googlebot attempts to crawl but receives errors or catastrophic response times, it's technical. If Googlebot doesn't even try, it's a quality signal.

Also check the type of URLs involved: thousands of facet filter pages? Architecture problem. Recent product sheets with unique content? Dig into server-side issues.

• Discovered not crawled is not a Google bug — it's a verdict on your content or infrastructure
• Google prioritizes its crawl budget: it won't crawl what it judges as valueless
• A server logs audit allows you to distinguish technical refusal from editorial refusal
• A high proportion of this status should trigger a critical analysis of your site

Does this statement truly reflect what we observe in the field?

Yes, and it's brutal. We regularly observe sites with 60-70% of URLs discovered but not crawled — often poorly managed e-commerce sites that generate thousands of filter combinations, or WordPress sites that index anything via the sitemap.

What Gary Illyes doesn't say: Google can also deliberately place URLs in "discovered not crawled" to test the site's reaction. If you fix a technical problem, Googlebot returns — sometimes within hours. If you clean up low-quality content, the effect is slower but measurable.

Should you always be alarmed by a high discovered-not-crawled rate?

No. It depends on which URLs are affected. If they're old-school pagination pages, blog archives from 2008, or tracking parameters, it's better that Google doesn't crawl them.

The problem arises when these are your new product sheets, your strategic landing pages, or your fresh editorial content. Then you have a real issue — either Google doesn't find them relevant, or your server is struggling.

Can you force Google to crawl these URLs?

No. [To be verified] but field experience shows that requesting a crawl via Search Console on 500 discovered-not-crawled URLs changes nothing. Google returns when it deems it worthwhile — or never.

The only solution: fix the root cause. Improve content, optimize the server, clean up the architecture. Googlebot is not an on-demand tool, it's an algorithm that prioritizes according to its own economic logic.

What should you do concretely if you have a high discovered-not-crawled rate?

First, segment the URLs involved. Export the Search Console report, classify by type: products, categories, blog, facets, parameters. Identify patterns.

Next, cross-reference with your server logs for the same period. Is Googlebot attempting to crawl and failing? Or is it not even trying? If attempts + 5xx errors or timeouts, it's a server problem. If no attempts, it's a quality signal or crawl budget issue.

What mistakes should you absolutely avoid?

Never force indexation via sitemap of thousands of low-quality URLs hoping Google will crawl them. You make the problem worse: Google detects that you're massively offering it content it judges valueless, and this degrades the overall perception of your site.

Another classic mistake: deploying an undersized server for a product catalog of 50,000 references. If your average server response time exceeds 500ms, Googlebot will slow down its crawl — or even abandon certain sections.

How do you verify that your site is compliant and optimized?

• Analyze the discovered/crawled ratio over the last 3 months in Search Console
• Export URLs with "discovered not crawled" status and segment by page type
• Check your server logs: is Googlebot attempting to crawl these URLs?
• Measure average server response time (target: under 200ms)
• Identify URLs with no added value and block them via robots.txt or noindex
• Improve content on strategic non-crawled pages (uniqueness, depth, relevance)
• Optimize your internal linking to push priority pages