Does Googlebot really ignore your multilingual site's accept-language header?

Why does Googlebot crawl without an accept-language header?

Googlebot does not behave like a standard browser. Unlike a user whose browser systematically sends an accept-language header reflecting their language preferences, Google’s bot mostly crawls without this HTTP header.

When this header is sent, it defaults to 'en' (English). This deliberate approach aims to ensure that Googlebot accesses a stable and predictable version of your pages, without any language bias introduced by the server.

What is the risk for a site that detects language via accept-language?

If your server detects the accept-language header and automatically serves different content based on language, you create a major indexing issue. Google will only see the version served by default (often English, sometimes the default language of your server configuration).

The other language versions remain invisible to crawling. Your French, Spanish, or Japanese site may technically exist, but Googlebot will never access it if access depends on a header it doesn’t send. This creates a complete blind spot in indexing.

How does Google actually distinguish language versions?

Google relies on explicit signals: distinct URLs by language (subdomains, subdirectories, URL parameters), hreflang tags declared in HTML or the sitemap, and visible content on the page.

The accept-language header never enters the equation. It is a volatile, client-side data point that Google deliberately ignores to prioritize structural signals that the webmaster can control.

• Googlebot does not read accept-language in 99% of cases—it crawls “neutrally”
• Switching content via this header makes your language versions invisible to indexing
• Distinct URLs + hreflang are the only reliable signals for multilingualism
• A manual language selection banner ensures that all versions remain accessible to crawl
• The content served to Googlebot must be identical to what is served to a user without language preferences

Is this statement consistent with observed practices in the field?

Yes, and it serves as a brutal reminder for those who persist in believing that Google “understands everything.” In audits of multilingual sites, we regularly observe cases where only the English version (or the default server version) appears in the index. The rest? Never crawled, never indexed.

The confusion often arises from a misunderstanding of how HTTP works. Many frameworks (especially those focused on “user experience”) automatically detect accept-language and redirect the user. Convenient for UX, catastrophic for SEO if no distinct URL structure exists in parallel.

What nuances should be added to this official guideline?

Mueller talks about “almost never.” This “almost” leaves some margin for uncertainty—Googlebot can send 'en' in certain contexts. In practical terms? If your site switches from French to English as soon as it detects 'en', you lose the French version even in this minority scenario.

Another nuance: this rule applies to initial crawling and indexing. For JavaScript rendering or certain specific tests (Search Console, Mobile-Friendly Test), Google can simulate different contexts, but this is not the standard behavior of Googlebot in its classic indexing pipeline. [To be verified]: no official data on the exact frequency of sending the 'en' header.

In what cases could this rule still pose a problem?

If you use a CDN or reverse proxy that detects accept-language upstream of your server and serves differentiated cached versions, you are exposed. Even if you have distinct URLs, if the CDN overrides language detection, Googlebot may be stuck on a single variant.

Classic case: Cloudflare Workers or Lambda@Edge with poorly configured language routing rules. The bot arrives at /fr/, but the worker detects the absence of the accept-language header (or the value 'en') and serves the English version on the French URL. Result: duplicate content, chaotic indexing, loss of local relevance.

What should be done concretely for a multilingual site?

First step: distinct URLs by language. Subdomains (fr.example.com), subdirectories (example.com/fr/), or URL parameters (example.com?lang=fr)—the structure doesn’t matter, as long as it is stable and crawlable without a specific HTTP header.

Second step: correctly implement hreflang tags. Each page must declare its linguistic and regional variants, including self-references. Googlebot relies on these tags to understand the relationships between versions, independently of any HTTP header.

What mistakes should be absolutely avoided?

Never serve different content on the same URL based on accept-language. This is the classic trap of “smart” frameworks that detect the visitor's language. For Googlebot, the URL example.com/product must always return exactly the same content, regardless of the header.

Avoid 302 redirects based on accept-language as well. Googlebot will follow the redirect but will index the target (often the English version), leaving your other language versions orphaned. If you must redirect users, do it using client-side JavaScript after the initial HTML load—this way, Googlebot always sees the canonical version.

How can I check that my site complies with this logic?

Test your URLs with curl by explicitly removing the accept-language header: curl -H "Accept-Language:" https://example.com/fr/. The returned content should be identical to what is visible in a browser configured in French.

Also use Google Search Console, Coverage section, to check that all your language versions appear as indexed. If only the English version shows up, you likely have a server-side detection problem based on accept-language. Inspect the URL using the URL Testing Tool—the returned HTML should match the expected language, without depending on a header that Googlebot does not send.

• Structure the site with distinct URLs by language (subdomain, subdirectory, or parameter)
• Implement hreflang tags on all pages, including self-references
• Never serve different content on the same URL based on accept-language
• Offer a manual language selection banner instead of automatic redirection
• Test server rendering with curl without the accept-language header to validate content stability
• Check in Search Console that all language versions are crawled and indexed