Is it true that we should abandon domain sharding for HTTP/2 crawling?

What is HTTP/2 multiplexing and why is Google interested in it?}

HTTP/2 multiplexing enables the simultaneous transmission of multiple requests and responses over a single TCP connection. With HTTP/1.1, the browser or crawler had to open several parallel connections to load multiple resources simultaneously — hence the limit of 6 connections per domain in most browsers.

For Google, the stakes are twofold: reduce network load on the crawling infrastructure (fewer TCP/SSL negotiations) and crawl more effectively sites that host many resources. Multiplexing eliminates the phenomenon of "head-of-line blocking" that slowed down HTTP/1.1.

Why are only 10-20% of sites crawled using HTTP/2?

Google is deploying this feature in a gradual and controlled manner. Splitt does not provide details on the selection criteria: site size, crawl volume, server compatibility, or simple internal A/B testing.

This partial deployment suggests that Google is testing the stability and impact on its infrastructure before a full rollout. Some servers or configurations might also poorly handle HTTP/2, causing crawl errors.

Should we still use domain sharding?

Domain sharding involved spreading static resources (images, CSS, JS) across multiple subdomains (cdn1.example.com, cdn2.example.com) to bypass the limit of 6 HTTP/1.1 connections per domain. With HTTP/2, this technique becomes counterproductive.

Multiplying domains now forces the browser (or Googlebot) to open several distinct HTTP/2 connections, negating the benefit of multiplexing. Google does not explicitly state this, but domain sharding can even slow down crawling if Googlebot favors HTTP/2.

• HTTP/2 eliminates the need for domain sharding due to multiplexing over a single connection
• Google only crawls a minority of sites using HTTP/2 (10-20%), while the rest remains on HTTP/1.1
• The gradual deployment indicates a testing phase, not a general migration
• Domain sharding can become a hindrance if Googlebot favors HTTP/2
• No public criteria on sites eligible for HTTP/2 crawling

Is this statement consistent with real-world observations?

The partial migration to HTTP/2 for crawling aligns with Google's usual caution regarding infrastructure changes. Server logs confirm that Googlebot predominantly uses HTTP/1.1, even on HTTP/2-ready sites.

What’s surprising is the lack of communication about the selection criteria. Google does not disclose whether sites crawled using HTTP/2 are chosen based on size, sector, or server configuration. [To verify]: no public data allows one to determine whether a site will be crawled using HTTP/2 or HTTP/1.1.

What nuances should be considered regarding this announcement?

Splitt presents HTTP/2 as a clear advantage, but it omits several points. First, HTTP/2 does not directly improve ranking: it’s an optimization on Google's side, not a ranking factor. Second, not all servers handle HTTP/2 optimally.

Some shared hosting providers or poorly configured nginx setups can lead to timeouts or connection errors with Googlebot HTTP/2. If Google detects these errors, it may fallback to HTTP/1.1 — but we don’t know how long this takes or if it impacts crawl budget.

[To verify]: Google does not specify whether a site that is not crawled using HTTP/2 loses crawling efficiency, nor if switching from HTTP/1.1 to HTTP/2 increases the number of pages crawled per session.

Is domain sharding really obsolete everywhere?

For sites crawled using HTTP/2, yes, domain sharding no longer makes sense. However, if 80-90% of sites remain crawled in HTTP/1.1, abruptly removing it could reduce crawling parallelism and slow down indexing.

Specifically, a site with 50,000 pages and aggressive domain sharding (4-5 subdomains) could experience slower crawling if Googlebot remains on HTTP/1.1. The reverse is also true: if Google switches to HTTP/2 without the site consolidating its domains, the advantage of multiplexing is lost.

How can I tell if Googlebot is crawling my site using HTTP/2?

The only reliable method is to analyze server logs. Look for Googlebot requests and identify the protocol used (HTTP/1.1 or HTTP/2). Most nginx or Apache logs display this information in the request field.

If you don't have direct access to the logs, use a tool like Screaming Frog Log File Analyser or OnCrawl to filter Googlebot requests and verify the protocol. Be cautious: a site may be crawled using HTTP/2 in some sections and HTTP/1.1 in others.

Should I immediately remove domain sharding?

If your logs confirm that Googlebot predominantly crawls in HTTP/2, yes, consolidate your domains. Combine your static resources on a single domain (or use a single CDN) to maximize multiplexing.

If Googlebot remains on HTTP/1.1, don't touch anything for now. Keep domain sharding until Google fully rolls out HTTP/2. Monitor your logs monthly for any switch.

What mistakes should be avoided when migrating?

Do not abruptly remove subdomains without redirects. If you consolidate cdn1.example.com and cdn2.example.com into cdn.example.com, set up 301 permanent redirects to avoid 404 errors during the transition.

Also, ensure that your HTTP/2 server is correctly configured: Brotli compression enabled, server push disabled (often counterproductive), sufficient connection limits. A poor HTTP/2 configuration can cause crawl errors worse than HTTP/1.1.

• Analyze server logs to identify Googlebot's crawling protocol (HTTP/1.1 or HTTP/2)
• Remove domain sharding only if Googlebot predominantly crawls in HTTP/2
• Set up 301 redirects when consolidating domains
• Check HTTP/2 server configuration (compression, connection limits, timeouts)
• Monitor logs monthly for any switch to HTTP/2
• Test server load post-migration: HTTP/2 may increase the number of simultaneous requests