Why does Googlebot stubbornly recrawl your 404 pages?

Does Googlebot really have a soft spot for dead pages?

Google's logic is based on a simple observation: the web is volatile. A missing page can come back. A temporarily broken URL can be restored. Googlebot adopts an opportunistic strategy rather than a definitive one.

In practical terms, the bot does not immediately classify a 404 as permanently dead. It schedules spaced crawling attempts to check if the resource re-emerges. This frequency increases if new links appear pointing to the concerned URL—this is a signal that someone, somewhere believes that this page still exists.

What triggers these repeated recrawls?

Two main factors drive this behavior. First, the appearance of new backlinks: if a third-party site creates a link to your 404, Googlebot interprets this as a sign that the page could be coming back. Next, the past popularity of the URL: a page that previously generated a lot of traffic or links stays on the radar longer.

The bot is not foolish. It regulates its efforts based on the probability of the page's resurrection. An old 404 without new signals ends up being crawled less frequently until it almost fades into oblivion.

What impact does this have on crawl budget?

Each request to a 404 consumes crawl budget without indexing anything. On a small site, the impact is negligible. On a large e-commerce catalog with thousands of archived references, it can quickly become a burden.

Google does not provide specific figures on the portion of budget wasted, but field observations show that sites with 30-40% of 404s in their logs can see their crawl efficiency drop drastically. The bot spends time on empty pages instead of discovering your new strategic pages.

• Googlebot recrawls 404s to check for their potential return, especially when new links appear
• This behavior consumes crawl budget without providing immediate indexable value
• The intensity of recrawling depends on the past popularity of the URL and recent external signals
• Old 404s without new signals gradually become neglected
• On large sites, a high proportion of 404s can seriously degrade crawl efficiency

Does this statement align with field observations?

Yes, and it's quite reassuring to see Google admitting this officially. Log audits have confirmed this for years: Googlebot never completely lets go of a 404, especially if it has had a rich past. We observe spaced recrawls (weekly, monthly) on dead URLs for 18 months, sometimes longer.

What becomes interesting is when we correlate these recrawls with the appearance of new backlinks. Sites engaging in neglected link building—by obtaining links to pages that are now 404—fuel this unnecessary cycle. Google comes back to check, always finds an error, and resumes a few weeks later.

What nuances should be added to Mueller's statement?

Mueller remains vague on the exact frequency and the thresholds that trigger these recrawls. [To verify]: Google never specifies how many new links are sufficient to restart an intensive cycle or how long a 404 stays on the active radar. These parameters probably vary depending on the overall authority of the site.

Another blurry point: the treatment difference between a true 404 (page never recreated) and a temporary 404 that actually returns. Google claims to check “just in case,” but does not say if it learns from its mistakes. If a URL returns 404 for 3 years without interruption, the recrawl should logically become very rare. Data is lacking for a definitive answer.

In what cases does this logic pose problems?

On high-content turnover sites—media, seasonal e-commerce, classifieds—the volume of 404s naturally skyrockets. Thousands of dead URLs still accumulate external backlinks for months. The result: Googlebot spends its time checking corpses, detracting from active pages.

Another problematic case: poorly managed migrations. If you haven't properly redirected your old URLs and they continue to receive links, Google will stubbornly crawl them indefinitely. This is wasted crawl budget, while these visits could have been used to explore your new content.

What concrete actions can be taken to limit wasted crawl budget?

The first action: clean your broken backlinks. Use Search Console to identify error URLs that still receive external clicks. If these pages had value, redirect them with a 301 to a relevant equivalent. If they didn’t have value, contact the source sites to request the removal of the link or its update.

The second action: audit your internal linking. Tools like Screaming Frog or Oncrawl detect internal links pointing to 404s. Correct them immediately. Each internal link to an error invites Googlebot to waste a request. You create the problem, not Google.

When should the 410 code really be used instead of 404?

The 410 (Gone) code indicates to Google that the page is permanently dead and will never return. Theoretically, this should speed up the abandonment of recrawling. In practice, the effect remains marginal according to field reports—Google seems to treat 404s and 410s quite similarly in the medium term.

Use the 410 mainly for sensitive content that you want to disappear quickly from the index (products removed for legal reasons, outdated pages of high stakes). For everything else, a standard 404 suffices, provided it is not fed with internal linking or active backlinks.

How can you check if your site is overusing crawl budget on 404s?

Download your server logs for a minimum of 30 days. Filter Googlebot requests and calculate the proportion that hits 4xx codes. If you exceed 15-20%, you have a problem. Identify the most crawled error URLs and prioritize them—redirecting, correcting internal links, or disavowing poor backlinks.

Complement with Search Console: Coverage section, Excluded tab. The “Not Found (404)” URLs that appear with recent detection dates signal that Google is still recrawling them. Cross-check with your backlinks to understand why these pages stay on the radar.

• Audit your server logs to measure the proportion of crawl consumed by 404s
• 301 redirect old strategic URLs that still receive backlinks
• Clean all internal links pointing to 404 errors
• Contact source sites for broken backlinks for updates or removal
• Use the 410 code only for definitively removed high-stakes content
• Monitor Search Console to identify recently crawled 404s and act quickly