Does Googlebot really ignore the crawl-delay directive in your robots.txt?

What is the crawl-delay directive and why does it exist?

The crawl-delay directive was introduced in the robots.txt standard to allow webmasters to define a minimum delay between two consecutive requests from a bot. Its purpose is to protect fragile servers from being overloaded by overly aggressive crawling.

Initially supported by engines like Bing or Yandex, this directive has always been ignored by Google. The official statement confirms this unambiguously: Googlebot does not interpret it, period.

How does Google handle crawl frequency then?

Google utilizes a dynamic adjustment system based on the health signals of your infrastructure. If your server responds quickly without errors, Googlebot speeds up. If 5xx errors appear or response times lengthen, the bot automatically slows down.

This approach assumes that modern servers are capable of handling traffic spikes. Google believes that a fixed pause between requests is obsolete compared to a reactive mechanism.

Why does Google maintain this position?

The answer can be summed up in one word: efficiency. Google wants to crawl the web as quickly as possible without waiting for arbitrary delays. A well-configured server with CDN, cache, and scalable infrastructure doesn’t need a crawl-delay.

The problem? Not all sites have modern infrastructure. Small sites on shared hosting or undersized servers may suffer from this policy.

• Googlebot ignores crawl-delay: no exceptions to this rule
• The adjustment is automatic: based on actual server performance
• No direct manual control: you cannot enforce a slowdown via robots.txt
• Search Console remains your only lever: modifying the crawl rate for critical cases
• Infrastructure is key: slow servers are penalized by default

Does this statement reflect reality on the ground?

Yes, and it is consistent with what has been observed for years. Googlebot has never respected crawl-delay, even when webmasters set values of 5 or 10 seconds. Server logs confirm that the bot sends requests without regard to this directive.

However, the claim that "servers are dynamic enough" needs nuance. In practice, thousands of sites run on shared hosting for €5/month that cannot handle 20 requests/second. Google either underestimates this reality or prefers to ignore it.

Does automatic adjustment really work?

In most cases, yes. When your server starts returning 503 or 504 errors, Googlebot indeed reduces its intensity. I have observed reductions of up to 70% of crawl activity after a series of server errors across multiple clients.

The catch: this mechanism is reactive, not preventive. Google waits for your server to show signs of weakness before slowing down. In the meantime, your site may have already experienced an overload, potentially impacting real users. [To be verified]: Google does not communicate the exact thresholds that trigger a slowdown.

What concrete alternatives exist to control crawling?

The Search Console still allows (for how long?) adjustments to the crawl rate, but only downwards and in exceptional cases. This option is gradually disappearing from the interface, replaced by a message urging improvements to infrastructure.

Real solutions involve technical optimization: implementing a CDN for static resources, aggressively configuring server caching, monitoring response times. If your server can handle normal load, Googlebot has no reason to slow down — and you have no means to force it to do so.

What should you do if your server is suffering from aggressive crawling?

The first step: analyze your server logs to confirm that Googlebot is indeed responsible for the overload. Sometimes, other bots (Bing, SEMrush, Ahrefs) are more resource-intensive. Isolate the user-agent Googlebot and count requests per hour.

If Google's crawl is indeed problematic, focus on optimizing response times. Enable GZIP compression, optimize your database queries, reduce TTFB. The faster your server responds, the less time Googlebot will spend connected.

What mistakes should you absolutely avoid?

Do not block Googlebot in robots.txt hoping to reduce load. You will kill your SEO. Do not set rate limiting at the firewall too aggressively: there is a risk of temporarily banning the bot and slowing down the indexing of your new content.

Also, avoid believing that adding crawl-delay will change anything for Google. This directive only serves to control Bingbot, Yandex, or third-party crawlers. For Googlebot, it is invisible.

How can you check if your infrastructure is suitable?

Check the Crawl Stats section in Search Console. If you see spikes in server errors (5xx) correlated with increases in crawling, your infrastructure is undersized. The goal: keep the error rate below 1%.

Test load capacity with tools like Apache Bench or Load Impact. Simulate 50 concurrent requests and observe response times. If they explode beyond 2 seconds, invest in a server upgrade or migrate to scalable hosting.

• Analyze logs to isolate Googlebot requests and quantify the load
• Optimize TTFB, compression, and server cache to reduce crawl time per page
• Monitor 5xx errors in Search Console and immediately fix the sources
• Never rely on crawl-delay to slow down Google — invest in infrastructure
• Consider a CDN to relieve your origin server of static resources
• Document crawl patterns to anticipate spikes and adjust resources