Why does Google display empty pages even when your JavaScript site is working perfectly?

How do browsers and Googlebot handle robots.txt differently?

Modern browsers completely ignore the robots.txt file. When you test your site in Chrome, Firefox, or Safari, each JavaScript request to your APIs goes through unfiltered. That’s why your page correctly displays the list of products, customer reviews, or price data.

Googlebot, on the other hand, strictly follows the robots.txt directives before executing any script. If an API URL is blocked, the bot doesn’t access it — period. The rendering on Google’s side then fails to load dynamic data, leaving you with empty HTML in the index.

How does this blocking create ghost pages in the index?

Imagine an e-commerce site that loads its product listings via fetch('/api/products/12345'). If robots.txt contains Disallow: /api/, Googlebot downloads the initial HTML, executes the JavaScript... but outright blocks the API request.

The DOM therefore remains skeletal: no product title, no description, no price. Google indexes this empty shell. When you test it in your browser, you see the complete page and think everything is fine. This is the classic manual testing trap that doesn’t reflect the realities of crawling.

Why are so many sites unknowingly blocking their APIs?

Many robots.txt files are automatically generated by CMSs or frameworks with default “security” rules. Developers block /api/ thinking they are protecting their data or preventing unnecessary crawling.

Other times, it’s a historical remnant: the site was built using pure server-side PHP and then migrated to React/Vue/Angular without cleaning up robots.txt. The result: critical endpoints remain blocked even though they are now essential for client rendering.

• Browsers never check robots.txt — your manual tests always pass
• Googlebot blocks any API request listed in Disallow, even for JavaScript rendering
• A poorly configured robots.txt generates empty pages in the index despite a functional site
• The problem is invisible without testing via Search Console or a Google rendering tool
• Modern frameworks amplify this risk by increasing client-side API calls

Is this statement consistent with what we observe in the field?

Absolutely. We regularly see SPAs or Jamstack sites with catastrophic indexing rates — 30% of pages indexed while the sitemap lists 10,000. When inspecting via the Search Console URL Testing tool, the HTML rendering shows empty <div id="app"></div>.

The diagnosis? A Disallow: /api/ or Disallow: /_next/data/ in robots.txt. Developers do not think “SEO” when configuring these rules — they think security, performance, or they copy-paste a template. And it breaks indexing without anyone realizing it for months.

What nuances should be added to this rule?

First point: if your content is already present in the initial HTML (SSR, pre-rendering, progressive hydration), blocking APIs has less impact. Google reads server-side content, even if subsequent JavaScript enhancements fail. But it remains a risky game — certain elements (dynamic prices, stock, reviews) may be missing.

Second nuance: some blocks /api/analytics, /api/tracking, or /api/user-prefs have no SEO impact and can legitimately remain blocked. The issue is that we often see Disallow: /api/ rules that are too broad, blocking everything. [To be checked] on a case-by-case basis: each endpoint should be assessed for its role in visible rendering.

When does this rule not apply?

If you are using strict server-side rendering (Next.js getServerSideProps, Nuxt asyncData server-side, classic PHP), robots.txt blocks nothing since data is injected before the HTML is sent. Googlebot receives complete content without executing JavaScript.

Another exception: sites that load non-indexable content by design (member areas, carts, user preferences). There, blocking /api/ is intentional and has no negative SEO impact. But let’s be honest — most of the time, it’s a configuration accident, not a carefully thought-out strategy.

How can you check if your APIs are not blocked?

First step: open your robots.txt and look for any line containing Disallow: /api, Disallow: /_next, Disallow: /data or equivalent. If you find this, it's an immediate red flag.

Next, use the URL inspection tool in Search Console. Click on ‘Test URL live’, then ‘View crawled page’ > ‘More info’ > ‘JavaScript’. Compare the final rendering with your actual page in Chrome. If entire sections are missing (products, articles, data), you’ve found the culprit.

What mistakes should be avoided when configuring robots.txt?

Never block an entire path like /api/ without thinking. If you need to protect certain endpoints, list them individually: Disallow: /api/admin, Disallow: /api/user-settings. Let through what serves public rendering.

Another classic pitfall: poorly ordered cascading rules. If you write Disallow: /api/ and then Allow: /api/products, the order matters for certain bots. Google handles this correctly, but it’s best to avoid confusion — be explicit and minimalist.

What should you do specifically to fix this issue?

Identify all API endpoints essential for the rendering of your indexable pages. Create a list: /api/products, /api/posts, /api/categories, etc. Ensure none of these paths appear in a Disallow directive.

If you must block some APIs for security reasons, instead use server-side authentication (tokens, headers, strict CORS) rather than relying on robots.txt. This file is not a firewall — it’s a guideline for cooperative bots.

• Audit robots.txt and remove any Disallow: /api/ rule that is too broad
• Test JavaScript rendering via Search Console on 10-20 strategic pages
• Compare the HTML crawled by Google with the actual browser rendering
• List critical API endpoints and explicitly allow their crawling if necessary
• Set up an indexing tracking alert to detect sudden drops
• Document robots.txt rules and their justification in your SEO runbook