Why does it take Google 3 to 6 months to refresh large websites?

What does Google mean by "large site"?

Google doesn't provide any specific numbers — frustrating, as usual. A large site could refer to either an e-commerce site with 50,000 products or a media portal with 500,000 articles. What matters is the volume of indexable pages and the frequency of updates.

In practice, as soon as your site exceeds a few thousand active pages, you enter this category. Crawling then becomes a balancing act: Googlebot cannot crawl everything continuously, it must prioritize. This is where the concept of crawl budget becomes essential.

Why this balancing act between new content and refreshing?

Googlebot has a limited crawl capacity per site, determined by the server's technical health and the domain’s authority. Each visit consumes resources — bandwidth, computation, storage. Google must therefore choose: explore new URLs or revisit those already known to detect changes.

Without an explicit signal, the bot adopts a conservative strategy. It prioritizes pages that change frequently (news, in-stock product pages) and slows down on static content. As a result, an updated page without notice may wait several months before the bot visits again. And during that time, your optimized content remains invisible to ranking algorithms.

How do sitemaps influence this refresh process?

The XML sitemap acts as a priority signal. By indicating the <lastmod> tag with a recent date, you signal to Google that a page has changed. But beware: Google does not blindly crawl entire sitemaps. It checks historical consistency — if you mark all your pages as modified every day while they remain unchanged, the signal loses its value.

Dynamic sitemaps, automatically generated with actual modification dates, are the most effective. They can significantly shorten the refresh time for strategic pages. This is the difference between waiting 4 months and getting a recrawl in 48 hours.

• Crawl Budget: limited resource allocated by Google to each site based on its size, speed, and authority
• 3-6 Month Window: average time for complete refresh without signals — variable based on the site's historical update frequency
• Strategic Sitemaps: reliable <lastmod> tag = priority signal to speed up recrawl
• Algorithmic Trade-Off: Googlebot prioritizes high-value pages (traffic, links, expected freshness)
• Supplementary Signals: IndexNow, Search Console, fresh internal links can reduce waiting time

Is this statement consistent with what we observe in the field?

Yes and no. On sites with 100,000+ pages, it is indeed observed that some URLs are only recrawled every 4-5 months if they are buried in the structure. But to claim a fixed 3 to 6 month timeframe is misleading: it all depends on the expected freshness of the page. An active product page with stock variations will be visited several times a day. A static "About" page may wait 8 months.

What’s missing here is granularity. Google doesn’t disclose the criteria that determine recrawl frequency: internal PageRank, recent external links, user engagement, content seasonality. [To verify]: to what extent do behavioral signals (CTR, dwell time) influence crawl prioritization? Google will never explicitly state this, but tests show a correlation.

What are the practical limitations of this recommendation regarding sitemaps?

The sitemap is useful, but it’s not a magic wand. If your site suffers from structural issues — server response time >500ms, excessive click depth, orphan pages — a sitemap won’t compensate. Googlebot can read the file, see the <lastmod>, and still decide not to crawl immediately if the site is perceived as technically fragile.

Another point rarely mentioned: Google has ignored <priority> and <changefreq> tags for years. Only the modification date truly matters. And again, Google compares this date to its own logs: if you mark a page as modified while it is bit-for-bit identical to the previous version, you lose credibility.

In what cases does this 3-6 month timeframe not apply?

High authority sites (national media outlets, institutional sites) enjoy a much higher crawl budget. Some see their strategic pages recrawled every hour. Conversely, a penalized or very slow site may see its budget reduced to zero — even with a perfect sitemap.

Pages linked from the homepage or powerful internal hubs are recrawled much more often than average. If you restructure your internal linking to elevate a strategic page to 1-2 clicks from the homepage, you can divide the refresh time by 10. This is an underutilized technique.

How can you speed up the refresh of strategic pages?

The first concrete action: generate a dynamic sitemap that accurately reflects modification dates. Forget WordPress plugins that mark all pages as modified on every visit. Use a script that compares content (MD5 hash) and only updates <lastmod> if the content has genuinely changed.

Next, leverage IndexNow for critical updates. This protocol (supported by Bing, Yandex, and indirectly Google via partnerships) instantly notifies search engines that a URL has changed. Result: recrawl in a few hours instead of several weeks. This is particularly effective for e-commerce sites that update prices and stocks in real time.

What mistakes should be avoided to prevent wasting crawl budget?

Don’t let Googlebot get lost in endless facets (poorly managed product catalog filters) or user sessions with URL parameters. Every wasted crawl on a useless URL is a strategic page that won’t be visited. Use robots.txt and the noindex tag judiciously.

Another classic pitfall: chain redirects. If Googlebot has to follow 3 301 redirects to reach a final page, it consumes its budget three times faster. Clean up ruthlessly. And monitor server response times: beyond 300ms, Google automatically reduces crawl speed to avoid overloading your infrastructure.

How can I check that my site is being refreshed properly?

In Google Search Console, under the "Settings > Crawl Stats" section, check the graph of the number of pages crawled per day. If this number stagnates or drops without apparent reason, you have a problem. Compare with the volume of pages you publish or update each week.

Also, use the URL Inspection Tool to force a one-time recrawl. But beware: abusing this feature (more than 10-20 requests per day) can be counterproductive. Google detects patterns and may ignore requests if it deems them automated or lacking real added value.

• Dynamic Sitemap: generate with real modification dates, check historical consistency
• IndexNow: implement for critical updates (prices, stocks, news)
• Internal Linking: elevate strategic pages to 1-2 clicks from the homepage
• Technical Cleanup: eliminate chain redirects, endless facets, unnecessary parameters
• GSC Monitoring: track crawl volume, server response times, 5xx/4xx errors
• Regular Audit: identify high-potential pages that haven't been crawled in over 60 days