Should you really leave the robots.txt file unchanged during an SEO migration?

Why does Google emphasize stability in robots.txt so much?

The statement from Martin Splitt reminds us of a principle often overlooked in practice: a technical migration should never trigger a revision of crawling rules. The robots.txt file defines what engines can or cannot crawl, and these decisions are based on a prior editorial or technical strategy.

If you had blocked certain sections (filter facets, archives, internal search pages) before the migration, it’s because you had identified a crawl budget, duplicate content, or performance issue. Changing these rules mid-migration introduces an uncontrollable variable into an already risky process.

What justifies a robots.txt block before migration?

Classic reasons include: avoiding the indexing of low-quality content (tags, filters without added value), protecting sensitive data (client areas, APIs), or preserving crawl budget by excluding dynamically generated URLs without SEO interest.

These strategic decisions remain unchanged with the underlying technology. Whether you are moving from WordPress to Shopify or from an Apache server to Nginx, the URLs to exclude remain the same — only their technical structure evolves. Let’s be honest: many SEOs take advantage of a migration to 'start fresh', and this is often where things go wrong.

In what context can this rule be problematic?

Google's statement assumes that your initial robots.txt was relevant. But what if the old file contained errors — overly broad blocks, deprecated directives, poorly calibrated wildcards — then you are stuck. Should you really postpone corrections until after the migration?

Google does not provide a clear answer here. The instruction 'do not change anything' primarily applies to sites where the robots.txt was properly configured. For others, you must balance the stability of the migration process and the necessity of correcting critical errors before transitioning.

• A migration changes the technical structure, not the crawling strategy
• robots.txt blocks should be decided beforehand, not during the migration
• Correcting a faulty robots.txt during a migration adds an additional risk
• Auditing the robots.txt file before the project prevents unpleasant surprises
• Google assumes your initial configuration was relevant

Is this directive realistic in the face of complex migrations?

Google's position makes sense in theory, but it becomes problematic in real-world scenarios. Many sites migrate precisely because their SEO configuration was flawed. Asking to keep a problematic robots.txt means perpetuating mistakes to avoid disrupting crawl.

Specifically, if your old site mistakenly blocks entire categories or contains now-unnecessary directives, you have two options: correct them before the migration (and risk destabilizing an already fragile site) or correct them afterwards (and endure a period where the new site inherits the same limitations). Neither option is ideal. [To be verified] — Google does not specify how to handle such scenarios.

What contradictions do we observe in practice?

In practice, many successful migrations actually include a complete review of the robots.txt. The argument 'don’t change anything' overlooks that some platforms automatically generate unwanted URLs (Shopify with its filtered collections, WordPress with its tag archives) that did not exist on the old site.

And this is where it gets tricky. If you are migrating from a custom site to a CMS that massively generates new URLs, should you really wait 'until after the migration' to block these new URLs? Google’s response remains vague. In the field, SEOs who waited often saw their crawl budget explode before they could react.

When does this rule clearly not apply?

If your migration involves a radical change of structure (switching from an e-commerce site to a showcase site, completely overhauling the architecture), keeping the robots.txt unchanged makes no sense. The blocked URLs may not even exist anymore or correspond to abandoned features.

In such cases, it’s essential to map the old rules, identify which still remain relevant, and build a new file before the switch. Yes, this adds a step. But pretending that a migration is neutral with respect to crawl is a fiction in many projects.

What should you check before launching a migration?

Before any switch, audit your robots.txt line by line. Identify each Disallow and User-agent directive, and ask yourself if it still makes sense. Many files contain rules added years ago by service providers who are no longer around, with no documentation.

Use Google Search Console to check which URLs are currently blocked. Compare them with your migration plan: will these URLs still exist? Under the same structure? If the answer is no, you cannot just copy-paste the file.

How to manage changes in CMS or platform?

A platform change often generates new technical URLs: Shopify creates /collections/, PrestaShop generates /index.php?id_category=, Magento adds sorting parameters. These URLs did not exist on the old site, so your current robots.txt does not cover them.

You must anticipate these new patterns and prepare corresponding rules ahead of the migration. Testing on a staging environment is essential: deploy the new site in pre-production, crawl it with Screaming Frog or Oncrawl, and verify that your robots.txt properly blocks what needs to be blocked.

What mistakes should you absolutely avoid during the transition?

Never remove all rules 'just in case' thinking you can restore them later. You will open a broadway for Google's crawl to low-quality or duplicate content, and the damage can be lasting.

Conversely, do not block everything out of fear of making a mistake. Some SEOs place a temporary Disallow: / 'for the time being,' which essentially means deindexing the site. The robots.txt is not an on/off switch for managing visibility: use meta robots tags for that.

• Audit the current robots.txt at least 30 days before the migration
• Map the new technical URLs generated by the target platform
• Test the new robots.txt on a staging environment
• Compare current rules with the new architecture (matching or obsolete)
• Document each directive (why it exists, what it blocks)
• Prepare a rollback plan if a critical block appears post-migration