Do AJAX calls really consume crawl budget or not?

Why is there a distinction between indexing and crawl budget?

Google distinguishes between two processes: crawling (exploring URLs) and indexing (recording content). AJAX calls often trigger requests to JSON or XML endpoints that return raw data.

These endpoints can end up indexed if Googlebot discovers them and no directive prevents it. However, Martin Splitt clarifies that they do not impact the crawl budget—which implies that Google treats them differently from traditional HTML pages.

How can you block indexing without disrupting rendering?

The X-Robots-Tag header applies at the HTTP level, before the browser or Googlebot even parses the content. Therefore, you can return X-Robots-Tag: noindex on your AJAX endpoints without touching the JavaScript code.

The main page continues to consume these resources for its client-side rendering, but Google does not index the JSON file as such. It’s clean, transparent, and prevents polluting the index with out-of-context data fragments.

What’s the difference with robots.txt or meta robots?

The robots.txt blocks crawling—so Google never retrieves the resource, which can prevent JavaScript rendering from functioning. The meta robots requires that the resource is HTML, which is not the case for a JSON endpoint.

The X-Robots-Tag allows crawling for rendering but stops indexing. This is exactly what’s needed for AJAX calls: Google downloads, executes, but does not archive.

• AJAX resources can be indexed if no directive prevents it
• According to Google, they do not consume crawl budget
• The X-Robots-Tag header blocks indexing without affecting rendering
• The robots.txt blocks crawling, thus breaking JavaScript—avoid it
• The meta robots only works on HTML, not on JSON

Is this claim about crawl budget verifiable?

Google never publishes precise metrics on crawl budget—no quotas, no counters. Therefore, stating that AJAX calls 'do not harm' the budget is a non-falsifiable statement [To verify].

On high-volume sites (e-commerce, aggregators), we observe that multiplying AJAX endpoints can slow down overall crawling. It’s hard to tell if this is a side effect (server latency, JS timeout) or if these resources genuinely consume crawl despite the official statement.

Is the X-Robots-Tag always enough to prevent indexing?

Yes, provided that your server correctly sends the header on each HTTP response. If you go through a poorly configured CDN or reverse proxy, the header may be lost—and your endpoints end up indexed.

Another trap: if your endpoint returns HTML instead of JSON (wrong route, dev error), the X-Robots-Tag remains valid but you end up with ghost pages in the index. Always check the Content-Type alongside.

Should you always block indexing of AJAX endpoints?

Not always. If your endpoint returns useful structured content (e.g., a public API, product schema data), indexing it can make sense—especially if you want to appear in specific searches.

However, in 90% of cases, indexing a raw JSON file is counterproductive: zero semantic context, no user experience, and index pollution. Let’s be honest: if you don’t know why you would want to index an endpoint, block it by default.

How can you detect incorrectly indexed AJAX endpoints?

Use the command site:yourdomain.com filetype:json in Google (or filetype:xml for XML endpoints). You’ll immediately see if any data files are indexed.

Additionally, inspect your server logs to spot Googlebot requests to /api/, /ajax/, /data/ routes. If these routes are not meant to be public, check that they properly return the X-Robots-Tag.

What is the proper server configuration to block indexing?

On Apache, add this to your .htaccess or vhost configuration:
<FilesMatch "\.(json|xml)$">
Header set X-Robots-Tag "noindex"
</FilesMatch>

On Nginx, in the relevant location block:
location ~* \.(json|xml)$ {
add_header X-Robots-Tag "noindex";
}

On modern frameworks (Next.js, Nuxt, etc.), configure the header directly in the API middlewares to ensure it is returned with each response, regardless of HTTP status.

What mistakes should you absolutely avoid?

Never block AJAX endpoints in robots.txt if your JavaScript relies on them for rendering. Google will crawl the page, try to execute the JS, fail to retrieve the data, and leave you with incomplete rendering.

Avoid mixing X-Robots-Tag: noindex and X-Robots-Tag: nofollow on the same resource without a clear reason. Nofollow only makes sense if your JSON contains links—which is rare. A noindex alone suffices in 99% of cases.

• Audit your site with site:yourdomain.com filetype:json to detect indexing leaks
• Implement the X-Robots-Tag on all AJAX endpoints that should not be indexed
• Check that the header is returned properly (curl -I or DevTools Network)
• Never block these resources in robots.txt if JS consumes them
• Test rendering in Search Console after modification to avoid regressions
• Document the configuration in your runbook to ensure the dev team doesn’t break it during refactoring