Does Googlebot actually ignore your forced HTTPS redirects?

What is HSTS and why do browsers use it?

HTTP Strict Transport Security (HSTS) is a security mechanism that forces browsers to always access a site via HTTPS. Once a browser visits a site with HSTS enabled, it remembers this rule and refuses any subsequent HTTP connections — even if the user types http:// in the address bar.

Browsers also maintain preloaded HSTS lists that contain thousands of domains that have requested to be forced to HTTPS on first visit. Chrome, Firefox, and Safari embed these lists directly. As a result, for these domains, the browser never sends an initial HTTP request.

How does Googlebot behave regarding HSTS?

Googlebot operates differently. It retains no HSTS list between crawls. Each URL is crawled with a clean slate, as if the bot has never visited the site before.

Specifically? If Googlebot finds a URL in http:// in its index, in an external link, or in your XML sitemap, it will try to crawl it over HTTP. It will not automatically convert the URL to https:// before the request, unlike modern browsers that apply HSTS.

Why did Google make this technical choice?

The reason relates to crawl neutrality. Googlebot must be able to discover and index the web as it actually exists, with its redirects, configuration errors, and accessible HTTP URLs. If the bot applied HSTS, it would obscure significant configuration issues.

For instance, a site that forgot to set up its 301 redirects but relied on HSTS to enforce HTTPS would go unnoticed. Users with compatible browsers would access HTTPS just fine, but Googlebot — and older browsers — would encounter HTTP. Googlebot therefore tests the raw reality of the server, not a version optimized by the client.

• Googlebot does not store any HSTS rules between its crawl sessions
• HTTP URLs in links, sitemaps, or indexes are crawled over HTTP
• Your 301 HTTP → HTTPS redirects remain essential for SEO
• HSTS protects end users but does not influence bot behavior
• This approach reveals server configuration errors that HSTS would mask on the client side

Is this statement consistent with field observations?

Yes, and it can be empirically verified. Server logs regularly show Googlebot requests over HTTP on domains that are on the HSTS preload list. If you analyze your access logs, you will find hits from Googlebot on port 80 even after years of full HTTPS migration.

This reality contrasts with the behavior of monitoring crawlers (Pingdom, GTmetrix) which often do apply HSTS. Googlebot deliberately maintains a "first principles" approach — it wants to know what the server actually responds to a raw HTTP request, without presuppositions.

What are the implications for incomplete HTTPS migrations?

This is where the issue arises. Many sites that migrated to HTTPS neglected their HTTP redirects, counting on HSTS or the assumption that "nobody uses HTTP anymore." False. Googlebot uses HTTP as the default entry point for many URLs.

If your 301 HTTP → HTTPS redirects are missing, slow, or misconfigured (redirect chains, timeouts), Googlebot loses crawl budget. Worse: it may index outdated HTTP versions or encounter 404 errors where the HTTPS version works. I've seen sites lose 20-30% of their indexed pages after an HTTPS migration because the HTTP redirects were faulty — but invisible to users due to HSTS. [To verify] if your own configuration doesn't have this hidden flaw.

In what cases does this rule have the most impact?

Three critical scenarios. One: sites with a lot of old HTTP backlinks. Googlebot follows these links over HTTP, and if the redirect is broken, the SEO juice is lost. Two: XML sitemaps still containing HTTP URLs by mistake — Googlebot crawls them as they are.

Three: sites with coexisting HTTP and HTTPS versions (yes, this still happens in 2025, especially on intranets or forgotten subdomains). Without proper redirects, Google may index both versions and dilute the ranking. HSTS does not save you from this duplicate content — only server-side 301 redirects do the job.

What should you check immediately on your site?

First step: audit your HTTP → HTTPS redirects with a tool that simulates Googlebot (Screaming Frog in Googlebot mode, or curl with Googlebot user-agent). Do not rely on your browser — it applies HSTS and shows you a distorted version of the bot's reality.

Ensure that each HTTP URL returns a 301 Moved Permanently to its HTTPS equivalent, without any intermediate redirect chains. An HTTP → HTTP → HTTPS chain wastes crawl budget and dilutes the PageRank passed. Also test the response speed of these redirects — a slow server on port 80 slows down the entire crawl.

How to audit your server logs for problems?

Analyze your server access logs (Apache/Nginx access logs) by filtering for Googlebot user-agent and HTTP protocol. If you see 404s, 500s, or timeouts on HTTP URLs while their HTTPS versions function properly, you have an invisible problem on the user side.

Also, look for crawl patterns. If Googlebot spends a lot of time on HTTP URLs that all redirect, that's wasted crawl budget. Ideally, update your internal links and sitemap to point directly to HTTPS — this reduces server load and speeds up the crawl. But never remove HTTP redirects, even after years: external backlinks still predominantly point to HTTP.

What mistakes should you absolutely avoid?

Number one error: deactivating port 80 after an HTTPS migration thinking that HSTS is enough. Googlebot will attempt to crawl over HTTP, fail, and gradually deindex your pages. I've seen an e-commerce site lose 60% of its organic traffic in three months after closing port 80 "for security reasons."

Second error: configuring 302 (temporary) redirects instead of 301. Google eventually treats them as 301, but with a delay. Third error: neglecting subdomains or www/non-www versions in HTTP — Googlebot crawls all variants and can create duplicate content if the redirects do not cover all cases.

• Test all HTTP URLs with curl or Screaming Frog in Googlebot mode
• Ensure that HTTP → HTTPS redirects return a 301, not a 302
• Eliminate redirect chains (HTTP → HTTP → HTTPS)
• Update the XML sitemap to exclusively point to HTTPS
• Analyze server logs for invisible client-side HTTP errors
• Keep port 80 open with redirects, never close it completely