Is Googlebot really just crawling without making any indexing decisions?

Gary Illyes's statement formally separates two stages of Google's pipeline: crawling and indexing. Googlebot retrieves content, period. Indexing decisions fall to other components of the infrastructure.

This distinction overturns certain conventional wisdom — namely the illusion that optimizing crawl is enough to guarantee indexing. Let's be honest: many well-crawled sites simply aren't indexed regardless.

What is the concrete difference between crawling and indexing?

Crawling is the simple retrieval of content by Googlebot. It traverses URLs, fetches HTML, retrieves resources, executes JavaScript if needed. Nothing more.

Indexing is the analysis of retrieved content, its processing, storage in the index, and the decision to make it accessible or not in search results. This stage depends on many criteria: quality, duplication, relevance, canonicalization, robots directives, and more.

Why does Google insist so much on this separation?

Because too many SEO professionals still confuse crawlability with indexability. A site can be perfectly crawlable — clean robots.txt file, impeccable sitemap, flawless internal linking — yet still have its pages rejected from indexing.

Google wants to clarify that indexing problems do not stem from Googlebot. If your pages aren't indexed, look toward quality criteria, canonicalization, E-E-A-T signals, duplication, and similar factors.

Which systems decide on indexing after Googlebot?

Google deliberately remains vague about technical details. We know that components like Caffeine (indexing infrastructure), quality algorithms, spam filters, deduplication and canonicalization systems all play a role.

In practice, once Googlebot has passed, content goes through multiple layers of analysis before landing in the index. And that's where everything is really decided.

• Googlebot retrieves, it does not judge
• Indexing falls to other systems after crawling
• Optimizing crawl never guarantees indexing
• Indexing problems don't come from Googlebot but from quality and relevance criteria

Does this statement really change anything in practice?

Not really — for those already following Google's logic. The crawl/indexing separation has been known for a long time, but this official confirmation allows us to put an end to recurring confusion.

The problem is that many sites frantically optimize their crawl budget thinking it will solve their indexing problems. Spoiler: it won't. A site can be crawled 10,000 times a day and only index 10% of its pages.

What nuances should be applied to this statement?

Gary Illyes simplifies deliberately to clarify, but reality is more intertwined. Googlebot reports signals that indirectly influence indexing: load time, HTTP errors, redirects, availability of critical resources.

Saying that Googlebot makes no decisions is technically true, but it collects data that feeds downstream decision-making systems. An important distinction.

Why are some well-crawled sites never indexed?

Because quality trumps everything else. A site can be technically flawless — fast, well-structured, accessible — and still be denied indexing if the content is deemed weak, duplicated, or poorly relevant.

E-E-A-T criteria, spam signals, internal cannibalization, canonicalization issues, all of this happens after Googlebot's visit. And that's where it often gets stuck.

What should you concretely do following this statement?

Stop obsessing solely over crawl. Yes, you need efficient crawling, but it's only one step. Focus on indexing criteria: content quality, originality, thematic authority, E-E-A-T signals, absence of duplication.

Check Google Search Console, Coverage section (or Pages in the new interface). Pages that are crawled but not indexed appear clearly — that's where you need to investigate.

What mistakes should you absolutely avoid?

Don't bet everything on robots.txt optimization or XML sitemap refinement thinking it will solve your indexing problems. These files facilitate crawling, not indexing.

Also avoid confusing discovered pages with indexed pages. Google can discover a URL without ever indexing it — it's actually common on large sites with weak or duplicate content.

How do you verify your site complies with this logic?

Audit crawlability and indexability separately. On the crawl side: check server logs, robots.txt file, sitemaps, internal linking. On the indexing side: analyze content quality, canonical tags, meta robots, relevance and authority signals.

Use tools like Screaming Frog for crawling and Google Search Console for indexing. Cross-reference the data to identify pages that are crawled but excluded from the index.

• Audit server logs to understand Googlebot's actual behavior
• Fix crawl errors (4xx, 5xx, redirect chains, timeouts)
• Analyze the quality of non-indexed pages
• Verify canonical and meta robots tags
• Eliminate duplications and weak content
• Strengthen E-E-A-T signals on strategic pages
• Monitor Google Search Console regularly, Coverage/Pages section