Should you really keep the robots.txt open on domains redirected with a 301?

Why does John Mueller emphasize the importance of the robots.txt file for redirected domains?

When you acquire multiple domains and want to consolidate their authority on your main site, the standard practice is to implement 301 redirects. But here’s the catch: if the robots.txt on the source domain blocks access to Googlebot, the crawler won't even be able to discover these redirects.

The robots.txt blocking prevents the bot from reaching the initial URL. The result is no redirect tracking, no PageRank transfer, no authority consolidation. You lose the very purpose of the maneuver.

What does it mean to map old URLs to new corresponding pages?

Mapping means establishing a logical match between the old URL structure and the new one. It’s not about redirecting the entire domain to your homepage: each page of the old site must point to the most relevant page on the new domain.

This granularity preserves user experience and maximizes link equity transfer. A blanket redirect to the root dilutes SEO value and often leads to a catastrophic bounce rate for users arriving on a page unrelated to their initial intent.

Do 301 redirects transfer 100% of authority?

Google has long stated that 301s transfer all link equity without loss. In practice, field observations show generally solid transmission, but not always complete depending on the context.

Factors such as domain age, thematic relevance between source and destination, or the speed of Google's processing can influence the effectiveness of the transfer. Nevertheless, the 301 remains the officially recommended method for consolidating domains.

• Open robots.txt is mandatory: blocking access nullifies the effects of redirects
• URL-by-URL mapping: logical matching between old and new structure
• Permanent 301 redirects: the only type of redirect suitable for domain merging
• Maintain redirects: keep 301s active for at least 12 months, ideally indefinitely
• Crawl verification: check in Search Console that Googlebot can access old URLs and follow the redirects

Is this recommendation consistent with observed practices in the field?

Absolutely. Cases of botched migrations due to accidental robots.txt blocking are more common than one might think. Typically, a technical team inherits an old domain with an ultra-restrictive robots.txt from a time when the site was under construction.

No one thinks to check this file before implementing the 301s. Six months later, one wonders why organic traffic hasn't increased despite acquiring high-authority domains. The answer is often there: a "Disallow: /" that kills everything in its tracks.

What nuances should be added to this directive?

First point: this recommendation applies to domains you are actively consolidating. If you permanently abandon a domain without the intent to recover its authority, the issue of robots.txt becomes secondary.

Second nuance: be cautious with domains acquired that have a doubtful history. Some may have accumulated toxic backlinks or a suspicious link profile. In these cases, a prior analysis is necessary before blindly redirecting to your main site. [To be verified]: Google does not clearly indicate if a penalized domain also transmits its penalty via 301, but field feedback suggests absolute caution.

In what situations does this rule not fully apply?

If you manage an active portfolio of domains (satellite sites, thematic microsites) that you do not wish to merge, then the 301s are obviously not the solution. You maintain each domain independently with its own appropriate robots.txt.

Another scenario involves expired domains purchased for PBN (Private Blog Networks). This practice, tolerated within a gray area, relies on maintaining separate domains to avoid network footprint. Again, no 301 redirects to the money site.

What practical steps should be taken when acquiring a domain?

First step: audit the acquired domain before any manipulation. Check its history in the Wayback Machine, analyze its backlink profile (using Ahrefs, Majestic, Semrush), and ensure there are no manual penalties in Search Console if you have access.

Next, establish a detailed mapping plan. Export the list of indexed URLs from the old domain (using Screaming Frog on a sitemap or a complete crawl), identify pages with the most authority and backlinks, and then associate each with the most relevant page on your main domain.

What mistakes should be absolutely avoided?

Classic mistake: redirecting the entire domain to your homepage. This approach dilutes authority and creates a disastrous user experience. Visitors clicking on a specific link end up on a generic page unrelated to their initial search.

Second mistake: forgetting to check the robots.txt after implementing the 301s. As Mueller points out, a residual robots.txt blockage nullifies all effort. Worse yet: leaving an active noindex on old pages, which also prevents equity transfer.

How can you verify that the consolidation is working properly?

Use Google Search Console on the old domain (add it as a property if not already done). Monitor the index coverage reports: you should see the old URLs gradually shifting to a "Redirected" status.

On the new domain side, observe the evolution of detected backlinks and organic traffic on the destination pages of the 301s. If the transfer works, you should see a gradual increase in authority on these target pages within 3 to 6 months following implementation.

• Audit the history and backlink profile of the acquired domain before any redirects
• Create URL-by-URL mapping between the old and new domain, prioritizing thematic relevance
• Check and clean the robots.txt of the old domain to allow Googlebot access
• Implement permanent 301 redirects at the server level (not via JavaScript or meta refresh)
• Ensure there are no residual noindex or canonical tags on the old URLs
• Monitor Search Console for both domains to track how Google processes the redirects