Does Google really index only a fraction of the web because of storage costs?

Why does Google publicly admit its technical limitations?

Contrary to the image of an infrastructure without limits, Google acknowledges here that indexing has a real cost — hard drives, SSDs, memory, electricity, maintenance. This statement from Gary Illyes shatters the myth of an engine that would index everything by default.

The real insight: Google makes strategic indexing choices based on the probability that content will be searched. It's not about raw volume, but anticipated relevance.

What does this concretely change for a website?

If your content is not deemed "desirable" by Google — meaning: likely to generate clicks from search results — it may simply never enter the index. Even if your site is technically perfect.

This aligns with field observations: orphaned pages ignored, low-traffic-potential content excluded, entire sites overlooked despite regular crawling. Crawl budget does not guarantee indexation.

What signals does Google use to decide?

Google doesn't detail its exact criteria, but we can infer several axes: site popularity, content freshness, existing behavioral signals, thematic authority, internal and external links. Isolated content, without context, without links, without preexisting traffic has little chance of being prioritized.

• Google does not index all of the web, only what it deems potentially searchable
• Storage cost is a real economic factor that influences indexing decisions
• The technical ability to crawl content does not guarantee its indexation
• Sites must prove that their content deserves to be stored and served to users

Is this statement consistent with what we observe in the field?

Absolutely. For years, we've seen technically accessible pages never get indexed. Google Search Console is full of URLs marked "Crawled, currently not indexed" — a status that perfectly illustrates what Illyes is saying.

The nuance: Google doesn't say how much this storage costs, or what percentage of the web is actually indexed. [To verify] We lack official figures on the actual crawl/indexation ratio. External estimates vary greatly.

In what cases does this rule not really apply?

Sites with strong authority — national media, established brands, government websites — benefit from far greater tolerance. Their pages are indexed massively, even those with low traffic potential.

For small sites or new entrants, it's a different story. Each page must justify its existence in the index. Let's be honest: Google doesn't apply the same selectivity rules to everyone.

What about content that deserves to be indexed but isn't?

That's where it gets tricky. If your content is objectively useful but ignored by Google, you must create artificial signals of desirability for it: strategic internal links, external mentions, direct traffic, social engagement. Anything that proves there's demand.

What should you do concretely to maximize your chances of indexation?

First, prioritize ruthlessly. If you have 10,000 pages and Google indexes only 3,000, maybe 7,000 don't actually deserve to be indexed. Audit your content and delete or consolidate what adds nothing.

Next, focus your efforts on high-potential pages: dense internal linking to them, external mentions, regular updates, engagement signals. Google must understand that these pages are actively searched for or visited.

What mistakes should you avoid at all costs?

Stop believing that an XML sitemap guarantees indexation. Stop producing content in bulk without a distribution strategy. And most importantly, stop thinking Google has a moral obligation to index your site.

The classic pitfall: automatically generate thousands of product pages or fine-grained categories, then be surprised they don't get indexed. Google sees that as noise with no added value.

How do you verify your strategy is working?

Monitor the ratio between crawled and indexed URLs in Google Search Console. If the gap widens, it means Google considers your content non-priority. Also compare monthly evolution: a healthy site sees its indexation rate stable or growing.

• Regularly audit "Crawled, currently not indexed" pages and decide: improve, merge, or delete
• Strengthen internal linking to strategic pages Google is ignoring
• Remove weak or duplicate content that dilutes your crawl budget
• Create signals of user demand (direct traffic, external links, shares)
• Prioritize quality and specificity over page volume
• Monitor indexation rate evolution monthly in GSC