Why does Google take months to recrawl certain pages on your site?

What causes these variable recrawl delays?

Google assigns a limited crawl budget to each site, proportionate to its size, authority, and update frequency. On a large site, Googlebot has to make choices: which pages deserve frequent exploration, and which can wait.

Strategic pages (homepage, main categories, fresh content receiving traffic) are recrawled every hour or daily. Deep, stable, or infrequently visited pages can linger in limbo for weeks or even entire quarters.

Is the sitemap really effective in speeding up recrawls?

Mueller recommends submitting a targeted sitemap to encourage Google to revisit certain URLs. Specifically, this means creating thematic or temporary sitemaps that include only recently modified pages, rather than a global file listing 50,000 stable URLs.

This approach works best on press or e-commerce sites, where updates are frequent and signal to Google that a visit is necessary. On a corporate site with few changes, the impact remains minimal.

How does Google decide which pages to crawl first?

No one knows the exact algorithm, but several documented signals play a role: modification frequency, incoming organic traffic, depth in the hierarchy, quality of internal and external backlinks, loading time.

Content that attracts direct traffic or clicks from the SERP will be recrawled more often. An orphan page, slow to load, with no internal links or backlinks, may be ignored for months even if it is listed in the sitemap.

• Limited crawl budget: Google cannot explore all pages of a large site continuously.
• Priority to active pages: those that change often or generate traffic are recrawled quickly.
• Targeted sitemaps: focusing on recent or modified URLs in a dedicated sitemap may accelerate their processing.
• Freshness signals: content changes, link additions, and clicks from the SERP influence recrawl frequency.
• Forgotten deep pages: a URL five clicks from the homepage, with no external links, may wait several months for a new crawl.

Is this statement consistent with field observations?

Yes, largely so. Server logs show massive disparities in crawl frequencies: some e-commerce categories are visited every hour, while disabled product pages or blog archives may remain ignored for three months.

The recommendation regarding sitemaps has been well-known for a long time, but Mueller does not provide any numbers or guarantees. It is a soft suggestion, not a promise of acceleration. [To verify]: no public data shows that submitting a sitemap significantly shortens the recrawl time on a site that already manages its internal linking well.

What nuances should be added to this advice?

The sitemap is not a magic wand. If your page is slow, orphaned, or considered low-quality content, the sitemap will not change that. Google can read your XML file and deliberately decide not to explore the listed URLs.

Moreover, creating too many thematic sitemaps can complicate maintenance: if you have 15 different sitemaps and forget to update one, you create noise. It is better to have a clean global sitemap with reliable <lastmod> tags than a fragmented, confusing setup.

When doesn't this rule apply?

On small sites (fewer than 500 pages), Google generally explores the entire site within a few days. The crawl budget issue does not really arise unless the site is technically disastrous (chain redirects, 5xx errors, response times > two seconds).

Sites with a flat architecture and solid internal linking also reduce the problem: if all your important pages are two clicks away from the homepage and receive internal PageRank, Google crawls them more frequently, whether there's a sitemap or not.

What should you do to optimize recrawling?

Start by cleaning your sitemap. Remove all URLs with 3xx, 4xx, 5xx errors, canonicalized to another page, or blocked by robots.txt. A clean sitemap only contains indexable and useful 200 URLs.

Then, enable the <lastmod> tags in your sitemap and ensure they reflect reality. If you modify a product page, the date should be automatically updated. Google uses this signal to prioritize its visits.

What mistakes should be avoided on large sites?

Do not create a giant sitemap of 100,000 URLs of which 80% have not changed in two years. Google will crawl it, see that there’s nothing new, and space out its visits. Segment by theme or update frequency.

Avoid submitting redundant sitemaps: if you have a global sitemap AND category sitemaps listing the same URLs, you create confusion. Google may crawl the same pages twice and ignore other areas of the site.

How can I check if my site is being crawled properly?

Use the Search Console: the