Why does Google still crawl your deleted old URLs?

Does Google really crawl dead pages for years?

Yes, it is well documented. Googlebot periodically revisits URLs that return a 404 code, even after the content has been permanently deleted. The reason? These URLs have left a mark in the index: external backlinks, historical mentions, accumulated authority signals.

The engine keeps track of these URLs and occasionally checks if they are back online. This is not a bug; it's a deliberate mechanism to detect a potential restoration of content. Specifically, if you delete a high-authority page and then republish it six months later, Google should be able to rediscover it.

Does crawling these old URLs consume my crawl budget?

No. Mueller is clear: this crawl occurs at low priority. The resources allocated to crawling your active pages are not diverted to these dead URLs. Google clearly distinguishes between priority crawling (new pages, updates, important content) and opportunistic crawling (sporadic checks, historical URLs).

In your server logs, these requests do appear, but they do not warrant any urgent corrective action. If your site generates enough fresh content, the total crawl budget remains mostly allocated to live pages.

Should I block these URLs in robots.txt to clean up the logs?

That's a bad idea. Blocking a 404 URL in robots.txt prevents Google from noticing that the page no longer exists. Result: the URL remains indefinitely in the index with an uncertain status, instead of being properly deindexed.

Allowing the 404 to occur enables the engine to confirm the permanent disappearance of the content and, ultimately, to remove the URL from the index. Blocking the crawl artificially prolongs the ghostly presence of these pages. Counterproductive.

• Google crawls historical 404s for years if they had backlinks or importance
• This behavior is normal and intentional, not a malfunction
• The crawl occurs at low priority and does not penalize the budget allocated to active pages
• Blocking these URLs in robots.txt hampers proper deindexation
• The server logs reflect this traffic, but it requires no corrective action

Is this statement consistent with observed practices on the ground?

Absolutely. Log analysts have long noted that Googlebot revisits URLs deleted years ago. What often surprises is the duration of this persistence: some 404 URLs continue to receive requests five, six, or even ten years after their disappearance.

The key variable? The backlink profile. A URL with 50 quality external links will be crawled much longer than a page without any incoming links. Google clearly applies a cost/benefit logic: as long as there is a non-zero probability that the page may reappear, occasional crawling remains justified.

What nuances should be added to this assertion?

Mueller talks about low priority crawl, but does not quantify. What proportion of the total crawl budget? How many requests exactly? [To be verified]. Without figures, it's difficult to assess the real impact on very large sites (millions of pages) with a massive history of deleted URLs.

Another vague point: the definition of an

What should you actually do with this information?

First, don’t panic when you see 404 URLs in your server logs. If these pages were historically significant, it’s normal for them to be revisited. Focus on crawling your active pages: as long as your new content is being discovered quickly, everything is fine.

Next, ensure your HTTP codes are correct. A 404 should be a real 404, not a soft 404 ("not found" page served as 200). Google needs to formally acknowledge the disappearance of content to adjust its long-term crawling behavior.

What mistakes should you absolutely avoid?

Never block in robots.txt the URLs you want to deindex. This common practice is counterproductive: it freezes the URL in an uncertain state and delays its definitive removal from the index. Let the 404 express itself freely.

Also avoid massively transforming your 404s into generic 301 redirects to the homepage. Some do this to "clean up" logs, but it creates a chaotic signal: hundreds of disparate URLs redirecting to unrelated content. Google detects this pattern and may consider it disguised soft 404.

How to optimize the management of your deleted URLs?

If you delete a page with backlinks, ask yourself: is there equivalent content on the site? If so, redirect with a 301 to that page. If not, own the 404 and let Google naturally acknowledge the disappearance.

For migrations or redesigns, plan a comprehensive mapping of redirects. Each historical URL should point to its most relevant equivalent, not to a catch-all destination. Yes, it’s tedious on large sites, but it’s what preserves your accumulated authority.

• Analyze your server logs to identify the most crawled 404 URLs (strong backlinks = persistent crawl)
• Ensure your 404s return a true 404 code, not a soft 404 with code 200
• Never block these URLs in robots.txt — let the 404 speak
• When migrating, create precise 301 redirects to equivalent content, not to the homepage
• Monitor the proportion of the crawl budget consumed by 404s: if it exceeds 10-15%, audit your redirects
• For deleted pages without equivalents, own the 404 and do not create artificial redirects