Why does PageSpeed Insights show different scores than what Googlebot actually sees?

Are PageSpeed Insights and Googlebot the same thing?

No. PageSpeed Insights uses its own crawler, which is technically based on Lighthouse and Chrome infrastructure, not Googlebot. When you run a PSI test, Google fetches your page with a different user agent than the one it utilizes for indexing your content.

Specifically? If your robots.txt blocks certain resources to Googlebot (typically CSS files, JavaScript, or fonts), PageSpeed Insights will still have access. It will return a score based on a complete version of the page, with all styles and scripts loaded. But Googlebot will crawl a truncated version.

Why does this distinction pose a problem in SEO?

Because you are optimizing blindly. You run PSI, see green across the board, and think that Core Web Vitals are top-notch. However, Google, on its side, indexes a page without critical CSS or missing certain scripts, which can degrade the actual user experience measured by the engine.

The classic trap: blocking /wp-content/themes/ or /assets/ in the robots.txt out of fear for crawl budget or due to misconfiguration. PSI still loads everything, but Googlebot sees a broken page, poorly structured, with incomplete rendering. Your Search Console metrics then show degraded signals that do not match your PSI tests.

How can I tell if my site is affected?

Three warning signals: significant discrepancies between PSI and Core Web Vitals reports in Search Console, warnings in GSC about blocked resources, or pages that display poorly in the URL inspection tool (incomplete HTML rendering).

Check your robots.txt line by line. Any Disallow pointing to CSS, JS, or fonts directories is suspect. Google has recommended for years not to block any critical rendering resources, but this mistake remains common in poorly configured CMS or haphazard migrations.

• PageSpeed Insights does not go through Googlebot, so it bypasses any robots.txt restrictions you impose on the engine.
• An optimal PSI score does not guarantee that Google indexes a performant version of your page.
• Blocking CSS or JavaScript to Googlebot creates a gap between your tests and the crawling reality.
• The Core Web Vitals measured by Google rely on what Googlebot can actually load, not on what PSI sees.
• Always check the URL inspection tool to compare Googlebot's rendering with your local version.

Is this statement consistent with what we observe on the ground?

Absolutely. We regularly see sites with impeccable PageSpeed Insights scores but persistent Core Web Vitals alerts in Search Console. A typical case: a WordPress site that blocks /wp-includes/ or /wp-content/ as a precaution, thinking it protects sensitive files.

PSI loads the page without restriction and shows 95+ everywhere. However, Googlebot, encounters a 403 on critical CSS, renders the page in degraded mode, and reports catastrophic metrics. The client cannot understand why their optimizations do not reflect in the rankings when all their tests show green.

What nuances should we apply to this statement?

Google does not explicitly specify which user agent PSI uses or how it handles potential redirects or geographic variations. We know that PSI is based on Lighthouse infrastructure, but the technical details remain vague. [To verify]: does PSI respect robots.txt directives for other user agents or does it completely ignore the file?

Another point: Google talks about "crucial resources" without precisely defining what is crucial. In reality, everything that impacts initial rendering is crucial: inline or external CSS, deferred loading JavaScript, webfonts, even certain SVGs. But a practitioner must test on a case-by-case basis as the boundary is not documented.

In what cases does this rule not apply?

If your robots.txt is clean and does not block any rendering resources, the difference between PSI and Googlebot will be marginal. This is primarily an issue with legacy sites, poorly configured CMS, or migrations where a robots.txt was copied and pasted without auditing.

Also beware of sites with overly broad Disallow rules (like Disallow: /wp-content/), often inherited from outdated SEO recommendations. In such cases, PSI and Googlebot consistently diverge. If you have a modern site with a minimalist robots.txt, you will likely never see this inconsistency.

What concrete steps should I take to avoid this discrepancy?

First action: audit your robots.txt immediately. Open it, identify all Disallow lines that point to directories containing critical CSS, JavaScript, fonts, or images. If in doubt, temporarily comment out the line and test Googlebot's rendering via the URL inspection tool.

Then, use the "Test robots.txt file" tool in Search Console (Settings > Explorers > robots.txt section). Enter the URLs of your main CSS and JS files, and check that they are not blocked for Googlebot Desktop and Googlebot Smartphone. If a critical file is prohibited, remove the corresponding rule.

What errors should I absolutely avoid?

Never block /wp-content/themes/, /assets/, /static/, /dist/, or any directory containing rendering resources. This is a classic error inherited from old SEO practices where it was thought to protect content or save crawl budget. Today, it just breaks Googlebot's rendering.

Another trap: relying solely on PageSpeed Insights tests to validate optimizations. PSI is a good indicator, but it does not reflect crawling reality. Always validate with the URL inspection tool and the actual CWV data from Search Console, collected over 28-day rolling periods.

How can I verify that my site is compliant after fixes?

Once your robots.txt is cleaned up, force a reindexing of key pages via the URL inspection tool. Wait 48-72 hours, then compare the HTML rendering captured by Googlebot with your local version. If styles and scripts are loaded properly, you are good to go.

Then monitor the Core Web Vitals report in Search Console over 4 weeks. If the metrics improve and get closer to your PSI scores, it means Googlebot is finally seeing the same thing as your testing tool. Otherwise, there are likely other blocks or stray redirects remaining.

• Audit the robots.txt and remove any Disallow rule blocking CSS, JavaScript, fonts, or critical images.
• Test each resource URL via the "Test robots.txt file" tool in Search Console.
• Validate Googlebot's rendering with the URL inspection tool, comparing it with the local version.
• Cross-check PageSpeed Insights scores with actual data from the Core Web Vitals report (28-day rolling).
• Force reindexing of strategic pages after correcting the robots.txt.
• Monitor discrepancies between PSI and GSC over a month to confirm the issue has been resolved.