Why doesn’t Google index all the URLs in your sitemap?

Does Google automatically index everything submitted to it?

No, and this is a persistent misunderstanding among many clients. The sitemap is not an order, it is a suggestion. Google discovers URLs through the sitemap but decides on its own whether they deserve to be in the index.

Specifically, the bot crawls what you declare, analyzes the content, and then applies its quality and duplication filters. If a page looks too much like another already indexed, or if Google believes it adds no value for users, it remains in the "Discovered – currently not indexed" status in the Search Console.

What prevents an already submitted URL from being indexed?

Two main factors: duplication (either real or perceived by the algorithm) and the estimated value of the page. Duplication is not just about copy-pasting: two product listings with nearly identical descriptions, or filtered pages that change only a cosmetic parameter, are enough.

Perceived value is more subjective. Google examines the content, user signals if available, depth in the site structure, and thematic consistency. An orphan page, without internal links, with 50 words of generic text, even in the sitemap, will never pass the threshold.

Why is Google communicating about this now?

Because too many sites still believe that an XML sitemap is a magic hack to index everything. E-commerce sites are pushing 500,000 URLs into a sitemap, of which 80% are filtered pages with no value, and then they are surprised by the catastrophic indexing rate.

Google aims to recalibrate expectations: the sitemap aids discovery, especially for deep pages, but it does not replace a solid architecture and differentiated content. It is a crawl tool, not a "index everything" button.

• The sitemap is a crawl suggestion, not an indexing command.
• Duplication and low perceived value are the two main barriers to indexing.
• An inflated sitemap of weak pages dilutes the signal and harms crawl budget.
• The Search Console distinguishes between "Discovered" and "Indexed": monitor this delta.
• Architecture and internal links are more decisive than presence in the sitemap.

Does this statement align with field observations?

Yes, and it's even an understatement. On large sites, we often see 30 to 60% of the URLs in the sitemap remaining in "Discovered – currently not indexed". The causes? Always the same: technical duplication (facets, poorly managed filters, pagination), thin content, or pages buried six clicks from the homepage without any internal links.

Google does not explicitly state how it measures the "value" of a page, and that’s where it gets complicated. We know that user signals, internal PageRank, freshness, and thematic consistency play a role, but the weightings remain vague. [To verify]: it is impossible to precisely quantify the threshold below which a page is deemed too weak for indexing.

What nuances should we bring to this statement?

First, Google does not specify whether the duplication it speaks of concerns pure textual content or also structural signals (title tags, H1, similar intentions). Our tests show that two pages with 70% common text but distinct intentions can both be indexed if the internal linking clearly distinguishes them.

Next, the notion of "value" varies by sector. An e-commerce product sheet with 100 words may be indexed if it has backlinks, direct traffic, or engagement signals. A blog page of 100 words, never. Context and domain authority matter as much as raw content.

In what cases does this rule not really apply?

News sites and large media benefit from preferential treatment: their content is crawled and indexed almost in real time, even if some pages are lightweight. Google prioritizes freshness and editorial authority over traditional value criteria.

The same goes for sites with very high domain authority: a SaaS giant can index minimalist landing pages because the overall trust compensates. For a small site, the same pages would remain blocked. The fairness proclaimed by Google hides a reality: not all sitemaps are treated with the same scrutiny.

How to audit the non-indexed URLs from your sitemap?

First step: export from the Search Console the report "Pages" > "Not Indexed", filter for "Discovered – currently not indexed", and cross-reference with your XML sitemap. You will get the exact list of URLs that Google has seen but deemed insufficient.

Analyze these URLs by type: are these filtered pages? Product variations? Empty categories? Deep paginations? Identify the patterns. If 80% are e-commerce facets, you know where to act. Use a crawler (Screaming Frog, Oncrawl) to measure click depth, unique content, and the presence of internal links.

What priority actions can improve the indexing rate?

Remove from the sitemap all low-value URLs: filters without additional content, paginations beyond page 3, nearly identical product variants. Keep only pages with substantial content and distinct search intent.

Strengthen the internal linking of non-indexed strategic pages. A URL six clicks away from the homepage, even in the sitemap, is unlikely to be indexed. Bring it up to 2-3 clicks via contextual links from high internal PageRank pages. Enrich the content of pages deemed "thin": add distinctive blocks, FAQs, user reviews, anything that increases perceived uniqueness.

How to avoid wasting the crawl budget on unnecessary pages?

Use robots.txt and noindex tags surgically. Filtered pages, internal search results, sorting variants: all of this should be blocked from crawling or set to noindex, and removed from the sitemap. Google will then crawl more intensely those pages that have high value.

Segment your sitemaps by content type (a product sitemap, a blog sitemap, a category sitemap). This allows you to fine-tune monitoring of the indexing rate by type and identify bottlenecks. Finally, monitor crawl frequency in the Search Console: if it drops after a sitemap cleanup, it’s a good sign; it means Google is concentrating its resources on fewer but higher-quality URLs.

• Export and analyze "Discovered – currently not indexed" URLs from the Search Console
• Remove all low-value pages (filters, deep paginations, duplications) from the sitemap
• Strengthen internal linking to strategic non-indexed pages
• Enrich the content of "thin" pages with distinctive blocks
• Block crawling (robots.txt) or set noindex for pages without search intent
• Segment sitemaps by type for fine tracking of the indexing rate