Does Robots.txt really block the crawling of your images across all your domains?

Why is robots.txt linked to the protocol and hostname?

Googlebot treats each protocol + hostname combination as a distinct entity. Specifically, a robots.txt placed on https://example.com does not apply to http://example.com or https://images.example.com.

This logic stems from RFC 9309 which standardizes robots.txt: the file must be retrieved from the root of the scheme + authority. If you migrate from HTTP to HTTPS without updating the robots.txt for HTTPS, Googlebot may crawl URLs you thought were blocked.

What happens with a 503 code in robots.txt?

A 503 status indicates a temporary unavailability. Google interprets this code as a caution signal and suspends crawling until the server becomes available again.

Conversely, a 404 or 410 indicates the absence of a robots.txt file, which equates to total permission to crawl. A misconfigured 503 can thus paralyze your crawl without you realizing it.

Do subdomains inherit the rules from the main domain?

No. Each subdomain requires its own robots.txt. If you block /images/ on www.example.com, a bot can still access cdn.example.com/images/ without restriction.

This is a common trap during migrations or redesigns: teams forget that blog.example.com or shop.example.com need their own configuration. External CDNs (Cloudflare, Akamai) present the same issue if you don’t control their robots.txt.

• A robots.txt applies only to the protocol + hostname pair (e.g., https://example.com ≠ https://www.example.com)
• A 503 temporarily suspends crawling; a 404 equates to total permission
• Subdomains and external domains each require their own robots.txt file
• A HTTP → HTTPS migration requires checking the robots.txt on both protocols
• Third-party CDNs may expose your images even if your main domain blocks them

Does this statement contradict observed practices in the field?

No, it confirms what technical SEOs have documented for years. Audits regularly show sites with a strict robots.txt on www but a totally open CDN or subdomain.

What’s missing here is a clear position on CDNs with custom domains (like cdn.yourdomain.com vs cdn-1234.cloudflare.net). Does Google crawl both? What’s the priority? [To verify] based on your canonical and DNS configuration.

What does "temporarily" really mean for a 503?

Mueller provides no figures. Field observations suggest that Googlebot retries after a few hours to several days, with an exponential backoff. But this depends on your site's usual crawl frequency.

An accidental 503 on a high crawl budget site could cost thousands of unupdated pages in the index. If your server returns a 503 due to a temporary overload, Google might wait 48 hours before retrying. There is no official guarantee regarding this timeframe.

Is it necessary to duplicate robots.txt on all subdomains?

Not necessarily. If a subdomain hosts public resources with no SEO value (static assets, internal APIs), a permissive or absent robots.txt may suffice.

However, if you are using multiple subdomains to segment content (blog, support, shop), each must receive an appropriate configuration. The risk: forgetting a subdomain that exposes test URLs or non-canonical duplicates.

How can you verify that your robots.txt covers all your domains?

Use the Search Console for each property (main domain, subdomains, HTTP/HTTPS variants). Check that the robots.txt file for each entity matches your strategy.

Next, list all your active hostnames: CDNs, functional subdomains, old redirected domains. A Screaming Frog crawl in "URL list" mode can reveal third-party domains serving your images unrestricted.

What should you do if you detect an accidental 503?

Immediately correct the HTTP code and force a re-crawl via Search Console. A prolonged 503 can drastically reduce your crawl budget and delay the indexing of new pages.

Set up a monitoring alert (Pingdom, UptimeRobot, custom script) to be notified if robots.txt returns anything other than a 200. A brief 503 lasting 10 minutes goes unnoticed; a 503 lasting 6 hours can impact your visibility for days.

Is it necessary to block images on an external CDN?

It depends on your strategy. If your images are hotlinked without editorial context, Google might index them without associating your brand. A robots.txt on the CDN can limit this risk.

However, blocking an entire CDN can prevent Google from validating your Core Web Vitals if your LCP or CLS depend on images hosted there. Test the impact before blocking massively. If you are using a third-party CDN without access to robots.txt (e.g., shared service), consider using X-Robots-Tag headers on your image URLs.

• Audit all your hostnames (www, non-www, subdomains, CDNs) and check their respective robots.txt
• Set up alerts to detect an accidental 503 on robots.txt
• Test the crawling of your images via Search Console on each distinct property
• If migrating from HTTP to HTTPS, ensure that both protocols have a consistent robots.txt
• Document third-party domains (CDNs, APIs) serving your resources and their crawling policy
• Consider using X-Robots-Tag if you do not control the robots.txt of a third-party domain