How does Google really decide which pages to index?

What’s the difference between crawling, indexing, and index selection?

Many still confuse these three steps. Crawling refers to when Googlebot visits a URL. Indexing is the process of content processing: semantic analysis, signal extraction, temporary storage.

Index selection comes after. Google decides if this document will actually be available for queries. A page can be crawled, analyzed, and still be excluded from the final index — or placed in a secondary, less prioritized index.

What signals does Google collect before making a decision?

Gary Illyes remains deliberately vague. He speaks of "multiple factors" without naming them. However, we know that certain signals carry significant weight: perceived content quality, duplication (exact or near-duplicate), UX signals, freshness, domain authority.

Other criteria are less obvious. The thematic coherence with the rest of the site, link depth from the homepage, the number of internal links pointing to the page, the presence of E-E-A-T signals — all contribute to an internal scoring that determines if the page crosses the indexing threshold.

Why does this selection exist?

Google cannot index the entire explorable web. The cost of storage, processing, and ranking an unlimited index would be prohibitive. Therefore, sorting is necessary — and Google prioritizes documents it deems useful to users.

This selection also serves as a defense against spam. Millions of automatically generated, duplicated, or empty pages are crawled every day. If all of them entered the index, the quality of the results would collapse. The selection is both a qualitative and quantitative filter.

• Crawling ≠ indexing: a visited URL is not necessarily stored.
• Multiple signals influence the decision: quality, duplication, authority, UX, thematic coherence.
• Necessary filtering: Google cannot and does not want to index everything it discovers.
• No guarantee: even a technically perfect page can be excluded if the signals are weak.
• Opaque process: Google rarely communicates about the exact thresholds or weights.

Is this statement consistent with real-world observations?

Yes, and it even confirms what SEO practitioners have been noticing for years. Sites with thousands of crawled URLs but only a fraction indexed, pages "Explored, currently not indexed" in Search Console — all of this can be explained by this active selection.

What remains frustrating is the lack of granularity. Gary Illyes mentions "multiple factors" without prioritizing them. It's impossible to know if thin content weighs more than a lack of internal backlinks, or if loading speed influences this sorting. [To be verified] in your own audits: correlate non-indexed pages with their Core Web Vitals signals, click depth, and Analytics bounce rate.

What nuances should be considered?

Google speaks of a "very sophisticated" process. Be careful not to overinterpret. Sophisticated does not mean infallible. Quality pages are sometimes wrongly excluded — especially on new sites, or in niches where Google lacks behavioral data.

The other nuance: this selection is dynamically. A page rejected today may be indexed tomorrow if the signals evolve: new backlinks, content updates, internal linking improvements. It’s not a definitive sentence, it’s a state at a given moment.

In what cases does this rule apply differently?

High-authority sites benefit from preferential treatment. A new article on a well-established media outlet will be indexed almost instantly, even if the content is light. In contrast, a recent or penalized site will undergo much stricter filtering — sometimes excessively.

Transactional pages (e-commerce product listings) are evaluated with different criteria than editorial content. Google is less tolerant of duplication on a category page than on a unique product listing. High-volume sites (millions of pages) must prioritize drastically: not everything can be indexed, and that is normal.

What should you do concretely to maximize your chances of indexing?

Prioritize quality over quantity. A site with 100 strong pages, well-linked, with original content and positive UX signals will be indexed better than a site with 10,000 generic or duplicated pages. Focus your efforts on high-value pages.

Work on internal linking: isolated pages, accessible in 5 clicks from the homepage, have little chance of overcoming the filter. Create contextual links from your main pages, use your blog articles to strengthen product or service pages. Internal PageRank remains a powerful signal.

What mistakes should be absolutely avoided?

Do not multiply low-value URLs. Poorly managed pagination filters, empty tag archives, indexable internal search pages — all of this dilutes your signals and consumes crawl budget for nothing. If a page provides nothing to the user, it should not be crawlable.

Avoid mass-generated content without supervision. Sites publishing 50 articles per day via AI, without human editing, linking, or promotion, often see a catastrophic indexing rate. Google detects the industrial production of weak content.

How can I check if my site is well-positioned for indexing?

Use the Search Console: section "Coverage" or "Pages" depending on the version. Identify the URLs "Explored, currently not indexed" and look for common patterns. Are they all deep in click depth? Do they have thin content? Are they lacking internal backlinks?

Compare the number of crawled pages (server logs) to the number of indexed pages (site:yourdomain.com). A ratio below 50% on a typical editorial site is a warning sign. On a large e-commerce site, a ratio of 30-40% may be acceptable if the key products are well covered.

• Audit the "Explored, not indexed" pages in Search Console and identify common causes.
• Strengthen the internal linking to non-indexed strategic pages.
• Eliminate low-value URLs: unnecessary pagination, duplicates, empty automatic content.
• Enrich the content of rejected pages: add unique text, media, structured data.
• Monitor the evolution of the crawl/indexing ratio over time — a sharp decline may signal a technical or quality issue.
• Test the impact of freshness: update a non-indexed page and see if it crosses the threshold after recrawl.