Why is the Vary header essential when serving content differentiated by user-agent?

What is the Vary header and why does Google care about it?

The Vary HTTP header indicates to caches and bots which request criteria influence the served content. When you return Vary: User-Agent, you signal that the same URL can produce different responses based on the requesting browser or device.

For Googlebot, this is a critical signal: the bot understands that it must crawl this URL with multiple user-agents (desktop, mobile) to capture all variations. Without this header, Google risks caching a single version and may never discover that your site serves different content to mobile devices.

When does this configuration still apply?

This directive targets sites that use server-side user-agent detection to serve different HTML versions at the same URL. This was common before the responsive design era: desktops received one structure while mobiles received another, without redirecting to a dedicated subdomain.

Today, most modern sites use responsive design (same HTML, adaptive CSS) or distinct URLs (m.example.com). In these cases, Vary: User-Agent is not necessary. However, if your backend infrastructure dynamically generates different HTML based on user-agent, this rule remains relevant.

What happens if you omit this header?

Without Vary: User-Agent, Googlebot may cache a single version of your page and assume it is identical for all devices. As a result, your mobile version may never be crawled correctly, or your desktop content may be indexed as the mobile version (and vice versa).

Google may also crawl your pages less frequently, assuming the content is static. The crawl budget is misallocated, and some strategic variations (enriched content for mobile, desktop-specific features) remain invisible to the search engine.

• Vary: User-Agent explicitly signals that the same URL serves different content based on the device.
• This directive only concerns sites with server-side detection, not pure responsive design.
• The absence of this header can cause mobile indexing issues and waste crawl budget.
• Modern sites with responsive designs or distinct URLs (m.example.com) do not need Vary.
• Google may cache the wrong version of your page if the header is missing.

Is this recommendation still relevant with mobile-first indexing?

Since the switch to mobile-first indexing, Googlebot primarily crawls with a mobile user-agent. If your site detects the agent and serves a lightweight mobile version, the absence of Vary might theoretically cause fewer issues than before. But be careful: Google continues to occasionally crawl with a desktop user-agent to check for consistency.

If you are serving substantially different content between mobile and desktop without Vary, you create a detectable inconsistency. Google may suspect cloaking or misinterpret your architecture. [To verify] how actively Google penalizes the absence of Vary, but reports show less effective crawls and mixed versions in the index.

Which sites should really worry about this directive?

Let’s be honest: the majority of sites launched after 2015 use responsive design and have never had to touch Vary. This directive mainly concerns legacy infrastructures (historic e-commerce, custom platforms, complex news sites) that have maintained a server-side detection logic for performance or advanced personalization reasons.

If you are dynamically generating different HTML depending on the user-agent (not just different CSS, but distinct markup), you are affected. AMP sites with canonical links to non-AMP URLs that detect the agent should also watch this point. On the other hand, if your backend sends the same HTML to everyone and only the CSS changes, you can ignore this recommendation.

What common mistakes are observed in the field?

The classic mistake: a site adds Vary: User-Agent while doing pure responsive design. Result: intermediary caches (CDN, proxies) store multiple unnecessary copies of the same page, degrading performance without benefit. This is an unjustified technical overhead.

The reverse is worse: a site serves genuinely different content but forgets the header. Google crawls the desktop version, caches it, and ignores the mobile version for weeks. Mobile-first indexing then relies on an unsuitable version. Technical audits often miss this point because tools rarely test HTTP headers with multiple user-agents.

How can you check if your site needs Vary: User-Agent?

The first step: test your site with different user-agents and compare the raw source HTML (not the visual rendering). Use curl with a desktop user-agent then mobile: curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" https://yoursite.com and curl -A "Mozilla/5.0 (iPhone; CPU iPhone OS 14_0 like Mac OS X)" https://yoursite.com.

If the returned HTML is strictly identical (excluding any dynamic timestamps), you do not need Vary. If entire sections change (structure, tags, content), you must implement the header. Do not rely on visual rendering: responsive CSS can hide differences that only exist on the presentation side.

How to correctly implement the Vary header?

On the server side, add Vary: User-Agent in the HTTP response headers. In Apache, use Header set Vary "User-Agent" in your .htaccess or your vhost. In Nginx: add_header Vary User-Agent;. In PHP: header('Vary: User-Agent');.

Be careful with multiple Vary values: if your site also varies based on Accept-Encoding or Cookie, combine them: Vary: User-Agent, Accept-Encoding. Do not overload Vary with irrelevant criteria: each added value multiplies the cached versions. Test with Chrome DevTools (Network tab, inspect Response Headers) to confirm that the header is present on all relevant pages.

What pitfalls should you avoid when implementing?

Do not confuse server-side user-agent detection with client-side feature detection. If your JavaScript adapts content after the initial load, the source HTML remains identical and Vary is unnecessary. The trap: some frameworks (Next.js, Nuxt) perform conditional SSR which may change markup based on user-agent. In that case, Vary is essential.

Another mistake: adding Vary without actually serving different content, just out of precaution. You unnecessarily fragment your caches and slow down the site. Conversely, do not remove Vary if your backend detects the agent, even if you think you will migrate to responsive "soon." By the time the migration is complete, you will have accumulated indexing issues.

• Test the source HTML with curl and multiple user-agents to confirm actual differences.
• Add Vary: User-Agent only if the HTML markup changes based on the agent.
• Check that your CDN respects the Vary header and does not ignore it in its cache configuration.
• Combine Vary with other relevant criteria (Accept-Encoding) without overloading the header.
• Regularly audit your HTTP headers using DevTools to confirm the presence of Vary.
• Document this configuration in your technical documentation to avoid regressions during updates.