Should you really create your robots.txt from scratch or can you take inspiration from a competitor?

Why does Google warn against copying robots.txt?

The temptation is great: you find a robots.txt from a competing site that seems well indexed, you grab it, make minimal adjustments, and deploy it. The problem is that this file reflects strategic choices that may not correspond to your technical reality.

An e-commerce site with 50,000 products and 200 facet filters doesn’t have the same needs as an editorial blog. Blindly copying can lead to blocking critical sections for your indexing — or conversely, allowing thousands of unnecessary pages to be crawled, which dilutes your crawl budget.

What are the common mistakes caused by this practice?

The most common case: blocking /search/ or /tag/ because another site does it, without realizing that your architecture makes these pages strategic linking hubs. Or the opposite: allowing /print/ and ending up with massive duplicate content in the index.

Another pitfall: overly generic rules like Disallow: /*? which block all URL parameters, including those used for internal tracking or some dynamic content enrichments. The result: Google only sees a stripped-down version of your pages.

How can you determine what exactly to block in your own robots.txt?

You need to start with a complete crawl audit: Screaming Frog, Botify, or OnCrawl depending on the size of the site. The goal is to identify URL patterns that generate noise without SEO value — infinite filter pages, user sessions, printable versions, exposed staging content, etc.

Then, cross-reference with the Search Console Coverage data: spot excluded pages, mass crawled 404 errors, permanent redirects that unnecessarily consume bot time. It’s this diagnosis that dictates the Disallow rules, not an external template.

• Every robots.txt must reflect the unique architecture of the site, not a generic model found elsewhere
• Copy-pasting can lead to unintentional de-indexing of strategic sections
• The crawl audit and Search Console analysis are the only reliable starting points for defining your rules
• A misconfigured robots.txt directly impacts crawl budget and the indexing of priority pages
• The maintenance of the file must follow the evolution of the architecture, not remain fixed on an initial model

Is this recommendation really followed by SEO practitioners?

Let's be honest: many sites still deploy copy-pasted robots.txt files from templates found on Stack Overflow or inherited from old migrations. The reflex of “it works for them, so it will work for us” is tenacious, especially in contexts where SEO is not a budget priority.

In audits, we regularly stumble upon absurd rules — WordPress sites blocking /wp-admin/ (normal) but also /wp-content/ (catastrophic for CSS/JS), or e-commerce sites banning /category/ while these pages carry all the internal linking. Proof that personalized reflection hasn’t taken place.

What nuances should be added to Google's statement?

Google doesn’t say you should never take inspiration from another robots.txt — it says you shouldn’t do it without reflection. An important nuance. Analyzing a competitor’s file can provide hints (“hey, they block their price filters, maybe we should do the same”), but always validate against your own situation.

[To verify]: Google never specifies how much a “poorly designed” robots.txt really impacts ranking. We know that wasting crawl budget harms the indexing of new pages, but the direct effect on positions remains hard to isolate in a controlled test. Caution is advised.

In what cases can this rule be relaxed?

For very simple showcase sites (5-10 static pages, no URL parameters, no dynamically generated content), a minimal — or even empty — robots.txt is more than enough. Copying a complex file in this context has no benefit and could even block resources unnecessarily.

Conversely, on complex platforms (marketplaces, aggregators, multilingual sites with subdomains), every line of the robots.txt must be thought out, tested, and documented. There’s no room for approximation: a mistake can lead to thousands of de-indexed pages without a visible alert for weeks.

What practical steps should you take to build your robots.txt?

The first step: map the site architecture by identifying URL typologies — products, categories, filters, internal search, user content, member areas, exposed APIs, etc. Then, for each typology, ask the question: should Google crawl and index this?

The second step: analyze the server logs to see what Googlebot is actually crawling. Often, we discover that it spends 40% of its time on infinite pagination URLs or session variants we thought we had blocked. It’s this delta between intention and reality that guides adjustments to the robots.txt.

What mistakes should be absolutely avoided when writing?

Never block / (the root) or critical CSS and JavaScript files for rendering — Google needs to execute JS to understand some modern pages. Always check with the robots.txt testing tool in Search Console before production.

Avoid overly aggressive wildcards like Disallow: /*.pdf if some PDFs are important SEO resources (white papers, guides). Prefer a granular approach: block only the specific directories that cause issues, not entire extensions.

How can you verify that the file works as intended?

Use the robots.txt testing tool from Search Console to validate each rule before deployment. Then monitor coverage reports for 2-3 weeks: if strategic pages suddenly disappear from the index with the status “Blocked by robots.txt”, it means a rule is too broad.

Complete this with a Screaming Frog crawl in Googlebot mode to simulate what the engine actually sees. Comparing it with a crawl without robots.txt (in “ignore robots.txt” mode) allows you to quantify the exact impact of each directive.

• Conduct a complete crawl audit before making any changes to the existing robots.txt
• Document each Disallow rule with its business or technical justification
• Systematically test with the Search Console tool before deployment
• Monitor indexing for 3 weeks post-deployment to detect any side effects
• Revisit the file every 6 months or after any major architecture overhaul
• Never block critical resources (rendering CSS/JS, strategic images)