Why does Google reserve the Crawl Stats report exclusively for domain properties?

What is the difference between a domain property and a URL prefix property?

In Search Console, Google offers two types of setup: domain property and URL prefix property. The former aggregates all variants of a domain (http, https, www, subdomains) into a single consolidated view. It requires DNS validation, proving that you indeed control the entire domain.

The URL prefix property, on the other hand, targets a specific URL: https://www.example.com or https://example.com/blog/. While easier to set up (HTML tag, Google Analytics, Tag Manager), it remains limited to the declared scope. If you have multiple subdomains or versions of the site, each prefix becomes a distinct property.

Why this restriction on the Crawl Stats report?

The Crawl Stats report analyzes Googlebot's behavior: number of requests per day, volume of data downloaded, average response time, returned HTTP codes. These metrics only make sense on the scale of the entire domain — an isolated subdomain or specific folder reflects only a fraction of the crawl.

Google has evidently decided that fragmenting this data by prefix would create more confusion than value. The domain property also imposes a DNS validation, ensuring that only the legitimate owner accesses strategic information about the site’s infrastructure. It's a security and consistency filter.

What data do we lose without access to the Crawl Stats report?

Without this report, you are navigating blind on several critical dimensions. It’s impossible to detect an abnormal crawl spike that could overwhelm your server resources. You cannot correlate a drop in performance with a sudden increase in bot requests. You can’t check if Googlebot is encountering recurring server errors or degraded response times.

You also lose visibility into the evolution of the crawl budget — how many pages Google crawls each day, which sections it prioritizes, where it spends unnecessary time. For a large e-commerce site or a media platform with thousands of pages, this blindness becomes crippling. Server logs remain an alternative, but their utilization requires time and technical skills.

• The Crawl Stats report is never accessible via a URL prefix property, regardless of the validation method used.
• The domain property requires DNS validation (TXT record), which centralizes all subdomains and protocols.
• Crawl data includes: requests per day, response time, volume of KB downloaded, distribution of HTTP codes, types of files crawled.
• For a site with multiple variants (http/https, www/non-www, subdomains), only the domain property provides a consolidated view of Googlebot's behavior.
• This restriction pushes SEOs to adopt a more rigorous validation architecture in Search Console.

Is this restriction really technically justified?

Let's be honest: Google could have perfectly offered limited crawl stats confined to the scope of a prefix. Technically, nothing prevents isolating Googlebot requests on https://example.com/blog/ and displaying specific metrics. However, the consistency of the data quickly becomes questionable — a bot does not always respect folder boundaries, it follows internal links and explores the domain as a whole.

The real reason likely lies in the simplification of the interface and security. By reserving these metrics for validated DNS owners, Google avoids fragmenting indicators that lose their meaning out of context. It also serves as a lever to push SEOs towards a complete domain setup — cleaner, more reliable, and less prone to perimeter errors.

What workarounds exist to analyze crawl without a domain property?

Server logs remain the go-to solution. Apache, Nginx, IIS — all log every request with IP, user-agent, URL, HTTP code, response time. Tools like Oncrawl, Botify, SEOlyzer, or custom Python scripts extract and aggregate this data. You then obtain a granular view of the crawl, much more detailed than Search Console.

However, this approach requires access to logs (not always obvious in shared hosting), a processing infrastructure (files can quickly weigh several GB per day on a large site), and skills to interpret the results. [To be verified]: some claim that third-party tools better capture multi-bot crawls (Googlebot Desktop, Mobile, Images, News) than Search Console, but Google has never confirmed any difference in granularity.

In what cases does this limitation pose a real problem?

For a monolithic site on a single domain, setting up a domain property is trivial. The SEO adjusts a DNS record, waits for validation, and gains access to the report. No drama. But on complex architectures — multi-brands, subdomains managed by different teams, white label sites — DNS validation becomes a bureaucratic ordeal.

Some organizations impose heavy IT processes to touch DNS. The SEO does not always have rights and has to go through internal tickets, waiting days. In the meantime, a crawl problem can devastate SEO performance. And that’s where the issue lies: Google imposes a technical prerequisite that, in certain structures, blocks access to critical data.

How to set up a domain property to access the Crawl Stats report?

Go to Search Console, click on the property selector in the upper left, then "Add Property". Choose "Domain" (not "URL Prefix"). Enter your domain without a protocol or www: example.com. Google will then generate a DNS TXT record to copy into your host or registrar's configuration.

Log into your DNS interface (OVH, Cloudflare, AWS Route 53, Gandi...), create a TXT record at the root of the domain (@) with the value provided. Be patient: DNS propagation can take a few minutes to several hours. Return to Search Console, click on "Verify". If everything is correct, the property validates instantly and aggregates all domain variants.

What mistakes should be avoided during setup?

The first classic blunder: adding the TXT record to a subdomain instead of the root. If you validate blog.example.com instead of example.com, only the subdomain will be covered. The Crawl Stats report will remain empty or incomplete. Always check that the record targets @ or the root, not www or any other prefix.

Another trap: some hosts automatically overwrite existing TXT records. If you already have a TXT for SPF, DKIM, or other, make sure your interface allows for multiple TXT values on the same domain. Otherwise, you risk breaking your email service by deleting the old record. Most modern DNS support multiple TXT — but confirm beforehand.

What to do if DNS access is blocked in your organization?

Negotiate with the IT team by explaining the SEO impact: without visibility into the crawl, it’s impossible to optimize the crawl budget, detect server overloads, or correct recurring 5xx errors. Prepare a data-driven argument: how many strategic pages are not being crawled, what is the potential loss in organic traffic.

If DNS remains locked down, switch to server log analysis. Request SFTP or SSH access to the log files (access.log, error.log), or set up automatic rotation to an S3 bucket that you will analyze with a third-party tool. It’s heavier, but often more granular than Search Console — you will see each bot request in real-time, not just a daily aggregate.

• Access your domain's DNS interface and create a TXT record at the root (@) with the value provided by Google.
• Verify that the record has propagated correctly (tool: mxtoolbox.com/TXTLookup.aspx) before clicking on "Verify" in Search Console.
• Confirm that the domain property aggregates all variants (http, https, www, non-www, subdomains) in the "Settings" tab of Search Console.
• If you manage multiple brands or independent subdomains, consider a domain property per brand to segment crawl reports.
• Document the DNS validation procedure in an internal wiki to prevent every team change from resetting the configuration.
• Set up an alert (Google Analytics, Tag Manager, monitoring tool) if the property loses its validation — this can happen during poorly managed DNS migrations.