Does Googlebot really send an accept-language header during crawling?

Why doesn’t Googlebot always send an accept-language header?

The behavior of Googlebot fundamentally differs from that of standard browsers. When a user opens Chrome or Firefox, their browser automatically sends an HTTP accept-language header reflecting their language preferences (e.g., "fr-FR,fr;q=0.9,en;q=0.8").

Googlebot, on the other hand, operates as a neutral agent. In most cases, it does not send any accept-language header, or uses English as the default value. This intentionally minimalist approach aims to crawl the most universal and accessible content possible, without linguistic bias.

The problem arises when a site relies exclusively on this header to determine which language to serve. The server detects the absence of a header (or "en-US") and automatically redirects to the English version — or worse, to an error page if no default language is configured.

What does this concretely mean for multilingual indexing?

If your multilingual architecture relies on automatic language detection via accept-language, Google will only see one version of your pages — typically the English one. The other language versions will never be crawled, hence never indexed.

You consequently lose all visibility in the localized SERPs. A site with French, Spanish, and German versions that automatically switches based on the header will only be indexed in English. Queries on Google.fr or Google.es will not surface your localized content.

This configuration also creates inconsistencies in hreflang tags. You declare alternative versions (via hreflang), but Google cannot crawl them to validate the matches. The engine then ignores your annotations, considering them to point to inaccessible resources.

Are there cases where Googlebot does send an accept-language header?

According to Mueller, such cases exist but remain exceptional. Google may sometimes send an accept-language header when crawling from Google Search Consoles configured for specific countries, or as part of internal testing of localized rendering.

However, these exceptions should never serve as the basis for an international SEO strategy. Relying on marginal and undocumented behavior risks having 95% of your multilingual pages ignored. The general rule remains: Googlebot does not communicate its language preference.

• Googlebot almost never sends an accept-language header, or uses English by default
• Sites that automatically switch content based on this header present Google with only one language version
• Alternative versions are thus neither crawled nor indexed, rendering hreflang tags useless
• The recommended solution: display a banner or language selector, without automatic redirection
• Distinct URLs by language (/fr/, /es/, /de/) remain the most reliable method for multilingual indexing

Is this statement consistent with real-world observations?

Yes, and it’s even a recurring problem among poorly configured multilingual sites. Server logs consistently show that Googlebot arrives without an accept-language header, or with "en-US" as the only value. Technical architects discovering international SEO often fall into this trap.

I’ve seen e-commerce sites lose 70% of their non-English organic traffic because they relied on automatic detection. The French, German, and Italian versions existed, were technically correct, but remained invisible to Google. The crawl budget focused on the English version, creating a total bottleneck.

Mueller's recommendation (to display a banner instead of redirecting) aligns with best practices documented for years. It is consistent with the official guidelines on hreflang and multilingual architecture. Nothing new, but a useful confirmation.

What nuances should be applied to this recommendation?

The language selection banner looks good on paper, but it poses user experience issues. Displaying a pop-up "Choose your language" every time a visitor arrives is intrusive. Users hate that, and so does Google if it’s poorly implemented (risk of being considered an invasive interstitial on mobile).

The pragmatic alternative: use distinct URLs by language (/fr/, /es/, /de/) and implement intelligent client-side geographic detection (via JavaScript), with storing the choice in a cookie or localStorage. The user arrives at the correct version without friction, while Googlebot freely accesses all URLs.

Another point: Mueller talks about "automatically switching content," but does not specify whether it concerns only 302/301 redirects or also server-side rendering with the same URL. [To verify]: If a site uses the same URL for all languages but changes the served HTML content based on accept-language, does Google index multiple versions? Probably not, but it deserves clarification.

In what cases could this rule be bypassed?

Technically, you could set up server-side rendering that detects Googlebot's user-agent and serves it all language versions through a prerendering system. But it's risky: Google often interprets such practices as cloaking.

The legitimate exception concerns sites that use subdomains or domains separated by language (fr.example.com, example.fr). In this case, the accept-language header does not come into play since each version has its own distinct entry point. Google crawls each domain independently.

What should be done for an existing multilingual site?

First step: audit server logs to check how Googlebot currently accesses your pages. Filter requests by the user-agent "Googlebot" and check which URLs are crawled. If you see only one language version in the logs, you have a problem.

Next, manually test by simulating a request without an accept-language header. Use curl or Postman to send an HTTP request to your site without this header, or with "en-US". Observe which version of the page is served. If you consistently land on English (or an error), your architecture depends on this header.

Technical solution: migrate to a distinct URL architecture. Implement paths like /fr/, /es/, /de/ (or subdomains). Each language version should have its own URL accessible without conditions. Configure hreflang tags correctly to link these versions.

What mistakes to avoid when setting up a multilingual structure?

Never automatically redirect users based on their IP geolocation or accept-language header via server-side 301/302 redirects. This is the most toxic setup for international SEO. Googlebot gets trapped in a loop or stuck on a single version.

Avoid pure JavaScript language selectors that modify content without changing the URL. Google can technically index JavaScript content, but it’s less reliable. If content changes dynamically without the URL reflecting this difference, you create conflicting signals for the engine.

Another classic mistake: implementing hreflang without checking that all declared URLs are actually accessible. Google tests hreflang matches by crawling the URLs. If your tags point to pages that redirect or block Googlebot, they are ignored.

How can I check if my site is correctly configured for Googlebot?

Use Google Search Console for each language version. Add each variant (example.com/fr/, example.com/es/, etc.) as a distinct property. Check that all receive impressions and clicks in their respective countries/languages.

Also test with the "URL Inspection" tool from Search Console. Request indexing of a page in each language and observe the HTML rendering returned by Google. If all pages display the same content (e.g., always in English), then automatic detection is interfering.

Finally, monitor the index coverage reports. If you declare 500 pages in French via hreflang but only 50 are indexed, it’s a red flag. Google probably cannot access the alternative versions properly.

• Audit server logs to identify which language versions Googlebot actually crawls
• Manually test requests without an accept-language header (via curl or Postman)
• Migrate to a distinct URL architecture (/fr/, /es/, etc.) if the site relies on automatic detection
• Implement hreflang tags correctly between all language versions
• Set up a Search Console for each language variant to track performance separately
• Use the "URL Inspection" tool to verify rendering of each version by Googlebot