Is blocking JS and CSS in robots.txt an SEO mistake or a legitimate strategy?

Why this clarification on robots.txt and JS/CSS resources?

For years, Google's recommendations have evolved chaotically on this issue. Initially, blocking CSS and JavaScript in robots.txt was a common practice — sometimes even recommended to save on crawl budget. Then Google changed its approach, insisting that these resources should be accessible.

Martin Splitt makes it clear: blocking these files is technically not cloaking, since you are not serving different content based on user-agent. But the effect is just as destructive if your content relies on these resources to be visible. Google crawls the raw HTML, loads allowed resources, executes JavaScript, applies CSS, and indexes the final rendering. If robots.txt blocks a critical resource, rendering fails — and your content disappears from the index.

What’s the difference between raw HTML and rendered HTML?

Raw HTML is what the server sends directly: tags, scripts, links to CSS and JS. Rendered HTML is what the browser (or Googlebot) displays after downloading, executing, and applying all these resources. If your main content is injected by JavaScript — typical of React, Vue, Angular frameworks — and that JS is blocked, Google only sees an empty shell.

This is exactly what happens when a key JavaScript file is disallowed in robots.txt: Googlebot cannot execute it, so the final DOM remains incomplete. The result: absent content, orphan pages in the index, wasted crawl rate on empty shells.

In what cases does this blocking really cause problems?

The problem arises as soon as your main content relies on blocked resources. A classic showcase site with content in the initial HTML and a bit of JS for animations? No drama. A full JavaScript site where all text, titles, images are injected by React? Total disaster if the JS bundle is blocked.

Even CSS can play a role. If you hide content with display:none or visibility:hidden in your initial CSS, then reveal it with JS, Google may never see it if either of the two resources is blocked. The engine indexes what it sees in the final rendering, not what is theoretically present in the DOM but invisible.

• Blocking JS/CSS via robots.txt is not cloaking, but produces the same effect if the content relies on these resources.
• Google indexes the rendered HTML, not the raw HTML — if rendering fails, the content does not exist for the index.
• JavaScript-heavy sites (React, Vue, Angular) are the most vulnerable to this type of blocking.
• Always check your robots.txt to ensure that no critical resource is blocked.
• Use the URL Inspection tool in Search Console to see exactly what Google renders.

Does this statement align with what we observe in the field?

Yes, and it's actually one of the few assertions from Google that perfectly matches practitioners' observations. I've seen dozens of sites lose 60-80% of their organic traffic after a poorly configured React overhaul, with a legacy robots.txt that blocked the main JavaScript bundle. Search Console showed indexed pages, but the inspection tool displayed an empty rendering — just the <div id="root"></div> with nothing inside.

The important nuance: Google does not say that blocking JS/CSS is systematically catastrophic. If your content is already in the initial HTML and JS is used only for user interactions (carousels, modals, forms), blocking won't impact indexing. But how many sites today still work like that? The majority of modern stacks inject everything dynamically.

Why doesn’t Google penalize this blocking as cloaking?

Technically, it's not cloaking because there is no intent to deceive. Cloaking is serving deliberately different content to Googlebot vs. users. Here, you are simply blocking access to a resource for everyone — Googlebot and theoretical users who would respect robots.txt (no browser does that, but let's move on).

That said, the end effect is identical: Google sees something different than your actual visitors. This is why some SEOs believe Google should be stricter on this point. But the official stance is clear: no manual penalties, just a natural indexing failure. The site shoots itself in the foot, no need for Google to step in.

What gray areas remain in this statement?

Martin Splitt does not specify how critical a resource must be to impact indexing. A JS file that injects 10% of the content? 50%? 100%? At what threshold does Google consider rendering to have failed? [To be verified] — we lack quantitative data on this point.

Another ambiguity: third-party resources. If you block an external CDN (Google Fonts, Bootstrap hosted elsewhere) which injects content via CSS ::before or ::after, will Google take that into account? Probably not, since this content is decorative, but the line remains blurry. Finally, what about subdomains? If your main JS is on cdn.yoursite.com and you block that subdomain in robots.txt, does Google make the connection? In theory yes, but in practice I've seen cases where indexing still worked — inconsistency among data centers?

How can I check if my site is affected by this problem?

First step: audit your robots.txt. Download it (yoursite.com/robots.txt) and look for all Disallow: directives targeting .js, .css, or directories like /assets/, /static/, /dist/. If you find blocks, list precisely which files are concerned.

Next, use the URL Inspection tool in Google Search Console. Enter a representative URL of each type of page (home, category, product sheet, article). Click on "Test URL live", then "View crawled page" > "Screenshot". Compare what Google sees with what a real visitor sees in Chrome. If the screenshot shows an empty or incomplete page, you have a rendering problem — likely related to a robots.txt block.

What mistakes should be absolutely avoided?

Never block a whole directory without knowing exactly what it contains. I've seen developers block /wp-content/ or /static/ "to save crawl budget" without realizing that all the critical JavaScript and CSS bundles were stored there. Result: massive deindexation in 48 hours.

The second classic mistake: inheriting an outdated robots.txt after a technical overhaul. You move from a classic PHP site to a React site, but keep the old robots.txt that blocked /js/ because at the time only analytics scripts lived there. Now, all the content is in /js/bundle.min.js — and it's blocked.

What can be done concretely to correct or prevent the issue?

If you identify a problematic block, remove it immediately from robots.txt and submit the updated file. Then force a reindexing via Search Console (URL Inspection > Request indexing) for your strategic pages. Google will take a few days to recrawl and restore the complete content.

To prevent the problem, integrate a robots.txt check into your deployment checklist. Before every production release, test the rendering of at least 5 representative pages using the Search Console tool. If you are using a modern JavaScript framework, explicitly document which files are critical for rendering and ensure they are never blocked.

• Download and audit the current robots.txt file to spot any blockages of .js or .css files.
• Use the URL Inspection tool in Search Console to compare Google’s rendering vs. actual browser rendering on 5-10 key pages.
• Immediately remove any blocks of critical resources and submit the corrected robots.txt.
• Add an automated rendering check in the CI/CD pipeline if possible (e.g., Puppeteer + diff screenshot).
• Document in an internal wiki which directories and files are critical for indexing and must never be blocked.
• Train developers and OPS to understand the SEO impact of robots.txt — it’s not just an innocuous technical file.