Does Googlebot really load your site as an anonymous user every time it visits?

Why does Googlebot not keep any cookies between visits?

Google wants its bot to explore the web as a completely neutral visitor, without any history or session bias. Each crawl starts from scratch: no tracking cookies, no session IDs, no cached data on the bot's side. This approach ensures that Googlebot sees the raw content that any user would see on their first visit.

Specifically, if your site displays different content depending on whether a visitor is 'new' or 'known' (via cookies or sessions), the bot will always see the 'new visitor' version. This poses a problem for sites that hide content behind light paywalls, poorly coded consent pop-ups, or session-based redirections.

Can prefetching influence Googlebot's crawl?

No. Prefetching (or preloading) involves loading resources before the user explicitly requests them — typically via <link rel="preload"> or <link rel="prefetch">. This technique does not speed up Googlebot: the bot loads pages in its own way, according to its own scheduling and crawl budget.

However, prefetching enhances the actual user experience by reducing perceived loading times. And if your Core Web Vitals improve because of this, Google might indirectly take that into account in ranking — but that's a side effect, not a direct lever on the bot.

What does this mean for sites that customize their content?

If you customize your content based on user history (recommendations, A/B tests, targeted offers), Googlebot will never see those variations. It retrieves the default version, the one an anonymous visitor would get. This can create a gap between what you optimize for humans and what Google indexes.

E-commerce sites that display different prices based on sessions, or SaaS platforms that hide certain pages from unconnected users, need to consider what Googlebot can really crawl. If strategic content is hidden behind a session, it simply won't be indexable.

• Googlebot does not retain any cookies: each visit is a total fresh start
• No persistent session state: impossible to 'recognize' the bot from one visit to another
• Prefetching does not speed up the crawl, but it can enhance UX and thus indirect signals
• Content personalized by session is not indexed: Google sees the default anonymous version
• Paywalls and cookie-based pop-ups can pose problems if poorly implemented

Is this statement consistent with on-the-ground observations?

Yes, and this has been confirmed for years by server logs. When analyzing Googlebot requests, we find that no cookies are transmitted in the headers, and no PHP session or equivalent is initialized. The bot behaves like a basic anonymous curl, with no state between visits.

Where things get tricky is with sites serving different content based on user-agent or IP. If you detect Googlebot and serve it a special version (light cloaking), Google may detect it through random tests from non-bot IPs. The lack of cookie retention does not prevent Google from verifying the consistency of the served content.

What nuances should be added regarding prefetching?

Mueller says that prefetching "can speed up the user experience, which is beneficial indirectly." This is true, but the SEO impact remains unclear. We know that Core Web Vitals count in ranking, but we don't know to what extent proper prefetching can compensate for mediocre content or a weak backlink profile. [To be verified]

Moreover, poorly configured prefetching can waste bandwidth and load resources that are never used. If it slows down the server or degrades the mobile experience, the effect can be counterproductive. As is often the case with Google, the recommendation is true in theory but lacks quantified thresholds.

In what cases could this rule be problematic?

Sites with dynamic or paid content are the most exposed. If you display a full article to new visitors but truncate the content after three visits (via cookies), Googlebot will always see the full version. This can create a gap between what Google indexes and what your returning users see — leading to high bounce rates if visitors do not find the promised content.

Another scenario: cookie-based A/B tests. If you test two page versions and Googlebot always lands on version 'A' (the default for anonymous users), your variant 'B' will never get indexed. Google recommends using server-side A/B tests with distinct URLs, but many marketers rely on client-side tools that depend on cookies.

What should you do to ensure Googlebot sees the right content?

The first step: test your site in private browsing mode, without any active cookies or sessions. This is what Googlebot will see. If critical elements (call-to-action, text blocks, images) disappear or change, that's a red flag. Also, use the URL Inspection tool in Search Console to verify the version rendered by Google.

Next, check your server-side personalization rules. If you display different content based on user history, ensure that the default version (the one for anonymous visitors) is complete and indexable. Don't hide your best pages behind authentication or a paywall without an alternative for crawling.

What mistakes should be avoided with cookies and sessions?

Never rely on a cookie to control access to strategic content. If your full blog article only appears after accepting an analytics cookie, Googlebot will not see it. The same goes for GDPR consent pop-ups: they must be non-blocking for the main content, else the bot might consider the page as empty or partially accessible.

Avoid redirections based on temporary sessions. If an anonymous user is redirected to a generic landing page but a known visitor accesses a detailed product page, Googlebot will only see the generic landing. Result: your product pages will never get indexed.

How to optimize prefetching without disrupting crawling?

Prefetching does not hinder Googlebot, but it can consume crawl budget on the server side if misconfigured. Prioritize preload for critical resources (CSS, fonts, hero images) and prefetch for pages likely to be visited next (probable navigation). Do not preload the entire site: it unnecessarily weighs down initial requests.

Use Resource Hints (dns-prefetch, preconnect) to speed up connections to third-party domains (CDN, analytics). This does not directly affect Googlebot, but it improves Core Web Vitals, which can influence ranking if everything else is equal.

• Test the site in private browsing to simulate Googlebot's view (zero cookies)
• Check the rendered version via URL Inspection in Search Console
• Ensure the 'anonymous' version of the site contains all strategic content
• Avoid blocking paywalls or pop-ups based solely on cookies
• Configure prefetching on critical resources without overloading the server
• Never hide indexable content behind authentication without a public alternative