Why does Google ignore the crawl-delay directive in robots.txt?

What is the crawl-delay directive and why does it exist?

The crawl-delay directive theoretically allows you to set a minimum delay between two consecutive requests from a web crawler. It is included in the robots.txt file with a simple syntax: Crawl-delay: 10 tells the bot to wait 10 seconds between crawling each page.

This directive was created to protect low-powered servers from being overwhelmed by too many requests. A small site hosted on limited infrastructure can quickly become saturated if Googlebot crawls 100 pages per second. Some engines like Bing or Yandex have adopted this directive, but Google has always refused to acknowledge it.

Why doesn’t Google support crawl-delay?

Google views this directive as too rigid and imprecise. A uniform delay does not take into account the technical reality of a site: some pages are light and quick to serve, while others are heavy and resource-intensive. Applying the same delay everywhere lacks granularity.

Google prefers an adaptive system that analyzes the server's response capacity in real-time. If the server responds quickly with 200 codes, Googlebot speeds up. If 503 errors or timeouts occur, it automatically slows down. This dynamic approach is seen as smarter than a static delay defined in robots.txt.

How does Google actually manage its crawl rate?

The engine uses several signals to automatically adjust its crawl budget. Server response time, error codes, timeouts, and even infrastructure performance signals are taken into account. Googlebot slows down if it detects that the server is struggling.

Webmasters have a tool in Google Search Console to limit the maximum crawl rate. This option is found in the old crawl settings (although Google recently simplified this interface). The adjustment allows you to cap the frequency but not to force it to increase.

• Google has never supported crawl-delay and probably never will
• Use Google Search Console to control the crawl rate, not robots.txt
• Googlebot automatically adapts according to the server response capacity
• Other engines like Bing and Yandex respect crawl-delay, creating inconsistency among bots
• A static delay does not fit Google’s adaptive logic

Is this position consistent with field practices?

On paper, Google's argument makes sense. An adaptive system that automatically slows down when the server shows signs of weakness appears smarter than a fixed delay. The problem is that webmasters lack visibility into this mechanism.

In reality, there are frequent instances where Googlebot overwhelms a server beyond its capacity, causing load spikes and 503 errors. The bot does slow down afterward, but the damage is already done. [To be verified]: Google claims that its system anticipates these problems, but server logs sometimes tell a different story.

What are the blind spots in this statement?

Google does not specify how it measures server capacity. Does it rely solely on HTTP codes and response times? Does it take into account CPU load on the server side? This lack of transparency frustrates professionals who would like to optimize their infrastructure accordingly.

Another point: the recommendation to use Search Console assumes that all sites have access to it. CDNs, complex multi-domain sites, or certain technical architectures make this control trickier than it appears. Moreover, the adjustment in GSC only allows limiting the crawl, not accelerating it.

Should you still use crawl-delay for other bots?

Yes, and this is where Google's position creates a practical inconsistency. Bing, Yandex, and numerous third-party bots respect this directive. A site that receives non-Google traffic (and many do) has a strong interest in setting crawl-delay to protect its infrastructure.

Some webmasters set different rules by user-agent: a crawl-delay for Bingbot, none for Googlebot. This approach works but complicates the maintenance of robots.txt. The ideal would be a universal standard, but we are far from it.

What should you do to control Google's crawl rate?

Forget the crawl-delay directive in robots.txt for Googlebot. It will simply be ignored. Focus on Google Search Console, under the "Crawl settings" section (or "Crawl stats" depending on the version of the interface). You will find crawl statistics there and, in some cases, the option to limit the rate.

Monitor your server logs to identify excessive crawl patterns. If you detect spikes that saturate your infrastructure, temporarily limit the rate via GSC. Keep in mind that this limitation slows down the indexing of new pages: it's a trade-off between server performance and indexing freshness.

How do you optimize your infrastructure to support Google crawl?

The real long-term solution is not to throttle Googlebot, but to make your site capable of handling the load. Implement a CDN to serve static resources, optimize database queries, and use aggressive caching for HTML pages.

From an architectural standpoint, isolate high-SEO-value pages from less important sections. Use robots.txt to block unnecessary sections (infinite facets, redundant URL parameters, back-office). The less time Googlebot spends on low-value content, the more efficiently it can crawl what matters.

What mistakes should you absolutely avoid?

Do not set a crawl-delay that is too aggressive thinking Google will observe it: it will ignore it, but you will penalize Bing and other bots. Do not confuse limiting the crawl rate with blocking URLs: robots.txt remains the tool for prohibiting access to certain sections, not for slowing down the bot.

Avoid also throttling the crawl by default in GSC without good reason. If your server handles traffic well, let Google explore at its natural pace. An optimal crawl budget speeds up the indexing of your updates and enhances your SEO responsiveness.

• Remove any crawl-delay directive for Googlebot in robots.txt (it is ignored)
• Use Google Search Console to limit the crawl rate if necessary
• Analyze your server logs for problematic crawl spikes
• Implement a CDN and caching to handle the load
• Block low-SEO-value sections via robots.txt (facets, redundant parameters)
• Maintain crawl-delay for the other engines (Bing, Yandex) if relevant