Where do these non-standard HTTP requests in my logs come from?

Most SEO professionals know that Googlebot primarily uses GET requests to crawl pages and occasionally POST for forms. However, server logs sometimes reveal HEAD, OPTIONS, PUT, PATCH, or DELETE requests coming from Google’s IP ranges.

Gary Illyes sheds light on this: these requests account for less than 1.5% of the total and are triggered by JavaScript code present on your pages. When Googlebot renders a URL, it executes the embedded JS. If this code contains fetch() or XMLHttpRequest calls with non-standard methods, Google actually executes them.

Why does Google execute this JavaScript instead of ignoring it?

The rendering process of Googlebot aims to understand the content as a modern browser would render it. Ignoring JavaScript would mean missing out on dynamically loaded content, client-side generated links, and critical interactions for user experience.

By executing the JS, Google mechanically triggers all network requests that this code generates. If your front-end application uses modern REST APIs with PUT for updates or DELETE for deletions, Googlebot will call them during rendering. The engine does not filter HTTP methods.

Should I block these non-GET requests in my server configurations?

No. Blocking HEAD, OPTIONS, or other methods could break the JavaScript rendering of your pages and prevent Google from seeing dynamically generated content. HEAD requests, in particular, are used to verify the existence of resources without downloading the full body.

The 1.5% proportion remains anecdotal in volume, but these requests can pertain to critical endpoints. If your JS queries an API to display primary content, blocking the method used would render that area invisible to Google. Keep your server configurations permissive for these methods, at least for Google user agents.

• Less than 1.5% of Google's total crawl volume uses methods other than GET and POST.
• These requests come from JavaScript executed during rendering of pages by Googlebot.
• The methods involved are HEAD, OPTIONS, PUT, PATCH, and DELETE.
• Blocking these requests can prevent correct rendering of your pages and hide content from Google.
• Server logs reflect the true behavior of JavaScript on your pages, not anomalies.

Does this revelation really change anything for SEO practitioners?

Let's be honest: most sites have either never noticed these requests in their logs or have not paid attention. The figure of 1.5% means that they remain marginal in volume, especially compared to the tens of thousands of daily GET requests on an average site.

The real question concerns modern web applications heavily using JavaScript frameworks (React, Vue, Angular). If your front-end triggers PUT or DELETE to APIs during the initial load, you need to ensure these endpoints respond correctly to Googlebot's requests. A 403 Forbidden or required authentication would block rendering.

What concrete risks are there for sites with complex REST APIs?

The issue arises when your JavaScript makes requests with side effects. A DELETE triggered by Google during rendering could theoretically delete data if your API does not have adequate protections implemented. [To verify]: Gary Illyes does not specify whether Google filters potentially destructive methods or executes them blindly.

A PUT or PATCH can modify the state of a resource. If your front-end uses these methods for real-time updates without CSRF tokens or origin checks, you potentially expose your data. The best practice remains to implement server-side guards: authentication, origin validation, rate limiting.

Are the data provided by Gary Illyes sufficiently precise?

The 1.5% threshold remains vague. We do not know the distribution by method type (HEAD vs DELETE, for example), the breakdown by site category, or if certain sectors are more affected. [To verify]: Does an e-commerce site with a lot of JS proportionally trigger more of these requests than a static blog?

Gary mentions that it is JavaScript that generates these requests, but he does not detail the execution context. Does Googlebot execute the event listeners (click, scroll) or only the code that executes on load? If Google simulates user interactions, the volume of 1.5% could rise on sites with a lot of interactivity.

What should you check in your server logs now?

Start by analyzing your logs to identify HEAD, OPTIONS, PUT, PATCH, and DELETE requests coming from Googlebot user agents. Cross-reference this data with your peak crawl times. If you notice suspicious volumes or unusual patterns, your JavaScript may be generating unintended calls.

Next, check the HTTP response codes returned for these methods. A high rate of 4xx or 5xx indicates that your server configuration or API is blocking these requests, which can prevent Google from rendering your pages correctly. Correct the firewall rules or Nginx/Apache configurations that reject these methods by default.

How can you protect your APIs from potentially destructive requests?

Implement a robust authentication system on all sensitive endpoints, even those called by your front-end. CSRF tokens, origin checks (Origin header), and rate limiting mechanisms help protect against unauthorized calls, whether from Googlebot or malicious actors.

For methods with side effects (PUT, PATCH, DELETE), require explicit permissions. Never rely solely on obscurity (security through obscurity). If Googlebot can trigger a DELETE, so can an attacker. Structure your APIs to require valid authentication for these operations.

Should you modify your front-end JavaScript to limit these requests?

Audit your JS code to identify unnecessary calls during the initial load. If your application sends PUT or DELETE during component mounting, ask yourself if these requests are really necessary to display the content. Delay non-critical operations until after user interaction.

Use conditional requests when appropriate. If-Modified-Since or If-None-Match headers allow your front-end to check the freshness of a resource without downloading the entire body. This reduces server load and optimizes crawl budget if Google executes these requests.

These technical optimizations can be complex to implement alone, especially on distributed architectures with multiple APIs and microservices. An SEO agency specializing in advanced technical audits can assist you in securing your endpoints, optimizing your JavaScript, and ensuring that Googlebot renders your content correctly without triggering unwanted side effects.

• Analyze server logs to identify HEAD, OPTIONS, PUT, PATCH, DELETE requests from Google.
• Check the returned HTTP codes for these methods and fix blocking configurations.
• Implement authentication and CSRF on all sensitive API endpoints.
• Audit front-end JavaScript to eliminate unnecessary calls during loading.
• Test the rendering of your pages with Google Search Console and Mobile-Friendly Test.
• Monitor crawl budget to detect any anomalies related to excessive JS requests.