How does Google determine which pages to crawl first on your site?

Where does the initial list of pages to crawl actually come from?

Google never starts from scratch. Each crawl session relies on a list of already known URLs, accumulated from previous visits. If Googlebot has already explored your product page three days ago, it will be on this list and may be recrawled.

The XML sitemaps constitute the second source of input. When you declare a sitemap in Search Console, you explicitly submit URLs that Google will add to its queue. This is an active signal, unlike the passive hope that an external link will eventually point to your new page.

What actually happens during a page visit?

Googlebot downloads the HTML code, executes (or not) the JavaScript as per the context, and extracts all the links present in the final DOM. Each discovered link feeds the crawl queue in turn — with priority depending on dozens of signals (PageRank of the source page, content freshness, link depth, etc.).

Waisberg's statement remains deliberately vague on these prioritization criteria. Google reads and follows links, indeed, but in what order? How frequently? No answers here.

Why is this mechanism critical for crawl budget?

If your site has 50,000 pages but your internal linking is chaotic and your sitemap is outdated, Google will prioritize crawling what it already knows — often the older, well-linked pages. New content or orphan sections may wait indefinitely.

This is especially visible on e-commerce sites with rapid stock rotation or media outlets publishing multiple articles daily. A product page that is never linked and absent from the sitemap can remain not indexed for weeks, even though it is technically accessible.

• The crawl list combines previous crawl history and submitted XML sitemaps.
• Discovered links on each page dynamically feed the crawl queue.
• No details are provided on the prioritization criteria for URLs in the queue.
• Orphan pages or those absent from the sitemap = high risk of delayed or nonexistent crawling.
• The XML sitemap is not a guarantee of indexing, but a signal for crawl consideration.

Is this statement consistent with what we observe in the field?

Yes, broadly speaking. Tests clearly show that pages added to the sitemap are generally crawled faster than those left to natural discovery via internal links. However, the actual speed depends on the crawl budget allocated to the site — a detail absent from Waisberg's statement.

It’s also observed that Google regularly ignores URLs present in the sitemap if they show negative signals: duplicate content, soft 404s, very low perceived quality. The sitemap is just a suggestion, not a mandate. [To be verified]: Google never publicly documents the relative weight of the sitemap vs. internal links in the prioritization algorithm.

What nuances should be added to this simplified view?

The statement completely overlooks the notion of crawl budget, which is crucial for large sites. Google doesn’t crawl indefinitely: it allocates a daily quota based on server velocity, domain authority, and perceived content freshness. To say 'Google visits and follows links' without specifying quantitative limits dodges the real question.

Another uncomfortable silence: no mention of JavaScript and rendering. Are links discovered after JS execution treated with the same priority as links present in the initial HTML? Field observations suggest longer delays, but Google never confirms this officially. [To be verified].

In which cases does this crawl logic fail?

Three classic scenarios where the process described by Waisberg is insufficient. First case: deep pages more than 5-6 clicks from the homepage, even if present in the sitemap. Googlebot rarely reaches them if the internal linking doesn’t regularly bring them to the surface.

Second case: sites with dynamic content generated by API or complex JavaScript filters. If links aren’t crawlable on the first pass, they never enter the discovery loop. Third case: recent or low-authority domains, where Google allocates such a meager crawl budget that it never exceeds the first 100 URLs, sitemap or not.

What should you do to optimize your site's crawl?

First reflex: audit the structure of your XML sitemap. Remove URLs marked as noindex, redirects, and 404 errors. A polluted sitemap dilutes the signal and wastes Googlebot's time. Update the modification frequency (lastmod) only when the content actually changes — a daily lastmod on static pages loses all credibility.

Next, strengthen the internal linking to your strategic pages. If a product category accounts for 30% of revenue but is six clicks from the homepage, Google won’t crawl it often enough. Create contextual links from pages that are already well-crawled (homepage, top categories, popular blog articles) to inject internal PageRank and shorten link depth.

What mistakes should be avoided to not waste your crawl budget?

Don’t let Googlebot get lost in endless filter facets (size, color, price, brand… combined into thousands of URLs). Use URL parameters in Search Console or robots.txt to block these paths. The same logic applies to excessive pagination pages: if you have 200 result pages, Google will crawl the first 10 and ignore the rest.

Another common trap: resources blocked in robots.txt that contain links. Googlebot cannot read the content of a blocked page, so it will never discover the links it contains. If you block /admin/, /tmp/, /cache/, ensure that no important page is exclusively linked there.

How can I check that Google is effectively crawling my site?

Check the Crawl Stats report in Search Console. You’ll see the number of pages crawled per day, the distribution by response type (200, 404, 301…), and the bandwidth consumed. If the number of crawled pages stagnates while you’re publishing 50 new articles per week, that's a red flag.

Compare the list of crawled URLs (server log files) with your sitemap and strategic pages. If Google spends 60% of its time on low-value pages /tag/ or /author/, your internal architecture needs to be revised. The server logs remain the source of truth: Search Console aggregates, logs detail each request from Googlebot with timestamp and user-agent.

• Clean the XML sitemap: remove 404s, redirects, and noindex pages.
• Strengthen internal linking to strategic pages to reduce link depth.
• Block filter facets and excessive paginations in robots.txt.
• Ensure that blocked pages in robots.txt do not contain critical links.
• Analyze server logs to identify crawled URLs vs. strategic pages.
• Monitor the Crawl Stats report in Search Console every week.