What does Google say about SEO? /
Quick SEO Quiz

Test your SEO knowledge in 5 questions

Less than a minute. Find out how much you really know about Google search.

🕒 ~1 min 🎯 5 questions

Official statement

To ensure that directives in a robots.txt are respected by all crawlers, it is advisable to firmly adhere to the standards, including the use of percent-encoding for special characters.
14:03
🎥 Source video

Extracted from a Google Search Central video

⏱ 9:15 💬 EN 📅 21/10/2026 ✂ 4 statements
Watch on YouTube (14:03) →
✂ Other statements from this video 3 ▾
  1. 5:14 How does Google really interpret each line of your robots.txt?
  2. 6:55 Why do emojis and UTF-8 characters break your robots.txt file?
  3. 11:46 Can robots.txt really block AI from scraping your content?
📅
Official statement from (26 days ago)
TL;DR

Google emphasizes strict adherence to the robots.txt standard, particularly the use of percent-encoding for special characters. Improperly encoded robots.txt can lead to directives being ignored by some crawlers, exposing critical sections or incorrectly blocking strategic resources. The stakes go beyond mere formal compliance: it's about effective control of crawling.

What you need to understand

What is percent-encoding in robots.txt?

Percent-encoding involves representing special characters by their hexadecimal code, preceded by the % sign. For instance, a space becomes %20, and an acute accent %C3%A9. This standard, defined in RFC 3986, ensures that URLs containing non-ASCII characters are interpreted uniformly by all user agents.

In robots.txt, this rule applies to path patterns declared after Disallow or Allow. If your directory contains URLs with accents, spaces, or Cyrillic characters, you must encode those portions. Otherwise, the crawler may interpret the pattern differently or even ignore it entirely.

Why is Google reinforcing this technical point now?

This statement likely comes after field observations of poorly formed robots.txt files. Many CMS generate this file automatically without strictly adhering to the RFC, especially for multilingual sites or legacy databases with non-normalized URLs.

The issue arises when a webmaster manually adds a directive for a path containing a special character. For example: Disallow: /données/ instead of Disallow: /donn%C3%A9es/. The crawler may then fail to recognize the pattern and crawl the section that was supposed to be blocked, wasting crawl budget or exposing unfinished content.

Do all crawlers uniformly adhere to this standard?

No, and this is precisely the crux of the message. Googlebot has strictly applied the standard for years, but other bots (Bing, third-party crawlers, scrapers) have variable implementations. Some tolerate raw characters, while others silently ignore them.

The result: a functional robots.txt for Googlebot may be bypassed by Bingbot or an AI crawler. This is why there is insistence on a firm adherence to the standard: it's the only guarantee for predictable multi-crawler interpretation.

  • Always encode non-ASCII characters in robots.txt path patterns
  • Strict adherence to RFC 3986 is the only guarantee of uniform interpretation by all crawlers
  • A poorly encoded pattern may be silently ignored, exposing critical sections or incorrectly blocking resources
  • Tolerance varies between crawlers: what works for Googlebot may fail elsewhere

SEO Expert opinion

Is this directive applied uniformly by Google itself?

Googlebot has indeed strictly adhered to the RFC 3986 standard since at least 2019. Tests show that patterns containing unencoded raw characters are treated literally: if the actual path is encoded in the URL, the raw pattern does not match. Conversely, an encoded pattern correctly matches the actual encoded path.

However, Google has long tolerated certain deviations in Search Console, displaying warnings rather than blocking errors. This statement marks a shift in tone: the advice is becoming almost mandatory to ensure expected behavior.

What are the real risks of a poorly encoded robots.txt?

The first risk is unintentional exposure of sensitive content. A real-life example: an e-commerce site blocked /panier/ with a raw pattern containing a tilde ~, which was misinterpreted by some crawlers. Result: tens of thousands of cart pages indexed, generating duplication and diluting crawl budget.

The second risk is accidental blocking of critical resources. A pattern that is too broad or poorly formed can prevent crawling of strategic categories. The last underestimated risk is incompatibility with new AI crawlers (ChatGPT, Perplexity) that apply the standard to the letter. [To be verified]: no public data on their tolerance rate for encoding errors.

In what cases can this rule be disregarded?

If your structure is 100% ASCII (URLs in English, without accents or special characters), the question does not arise. But as soon as a site involves multilingual aspects, legacy URLs with spaces, or complex GET parameters, encoding becomes essential.

Be cautious of false positives: some automated audits report encoding errors when the robots.txt is correct. Always manually check with the Test robots.txt file tool in Search Console, which strictly applies Google's parser.

If you manage a multilingual site with non-ASCII URLs or complex parameters, a thorough audit of the robots.txt is critical. An error can go unnoticed for months and cost thousands of poorly crawled pages.

Practical impact and recommendations

How can I check if my robots.txt respects the encoding standard?

The first step: run the file through the Search Console robots.txt tester. Manually enter URLs containing special characters from your directory and check that the behavior (allowed/blocked) corresponds to your intention.

The second method: use an online RFC 3986 validator to identify non-ASCII characters in your patterns. Any character outside the range [a-zA-Z0-9-._~:/?#[]@!$&'()*+,;=] must be encoded. A simple Python script with urllib.parse.quote() can automate the conversion.

What should I do if my CMS generates a non-compliant robots.txt?

Many CMS (WordPress, Drupal, Magento) dynamically generate robots.txt through hooks. If you detect an issue, switch to static mode: create a robots.txt file at the root of the server and disable dynamic generation. This gives you total control over the format.

For SaaS platforms where you lack direct file access (Shopify, certain builders), check for manual editing options for robots.txt. If no option exists, contact technical support with a clear reference to RFC 3986. A non-compliant robots.txt is a legitimate bug.

What common mistakes should I absolutely avoid?

The number one mistake: copy-pasting URL paths from a browser without checking the actual encoding. The browser often displays decoded characters for readability, but the underlying technical URL is encoded. Always compare with the raw address bar or the network inspector.

The second trap: using wildcards (*) without testing their scope with encoded characters. A pattern /cat*/ may not match /cat%C3%A9gorie/ depending on the implementation. Always test each critical pattern with real URLs from your site.

  • Validate the robots.txt in Search Console with real URLs containing special characters
  • Consistently encode any non-ASCII character with percent-encoding (e.g., é → %C3%A9)
  • Switch to static mode if the CMS generates a non-compliant robots.txt
  • Test patterns with wildcards on encoded URLs to verify matching
  • Automate the verification with a script validating RFC 3986
  • Document each directive for easier maintenance
Proper encoding of the robots.txt is not a minor technical detail: it is a direct lever of control over crawl budget. A poorly formed file can expose sensitive content or block strategic resources for months. These optimizations require a fine expertise of web standards and crawler behavior. If your infrastructure is complex (multilingual, legacy URLs, extensive directory), consulting a specialized SEO agency can ensure rapid and sustainable compliance, avoiding costly errors.

❓ Frequently Asked Questions

L'encodage percent-encoding est-il obligatoire pour tous les caractères spéciaux dans robots.txt ?
Oui, tout caractère hors de la plage ASCII standard (a-zA-Z0-9-._~:/?#[]@!$&'()*+,;=) doit être encodé en pourcentage pour garantir une interprétation uniforme par tous les crawlers. Cela inclut espaces, accents, caractères cyrilliques, chinois, etc.
Un robots.txt mal encodé peut-il pénaliser mon référencement ?
Pas directement via une pénalité algorithmique, mais indirectement en exposant du contenu non finalisé, en gaspillant du crawl budget sur des pages inutiles, ou en bloquant par erreur des ressources stratégiques. L'impact se mesure en pages mal crawlées et en dilution du PageRank interne.
Comment tester rapidement si mon robots.txt respecte le standard RFC 3986 ?
Utilisez le testeur robots.txt de Google Search Console en entrant des URLs réelles contenant des caractères spéciaux de votre site. Vérifiez que le comportement (autorisé/bloqué) correspond à vos directives. Complétez avec un validateur d'URL RFC 3986 en ligne pour détecter les caractères non encodés.
Les crawlers d'IA (ChatGPT, Perplexity) respectent-ils ce standard d'encodage ?
Les informations publiques sont limitées, mais les observations suggèrent une application stricte du standard. Un robots.txt mal encodé risque de ne pas être respecté correctement par ces nouveaux agents, exposant votre contenu à des usages non désirés.
Que faire si mon CMS ne permet pas d'éditer manuellement le robots.txt ?
Pour les plateformes SaaS fermées, contactez le support technique en référençant explicitement la RFC 3986 et en documentant le problème. Si aucune solution n'existe, envisagez une migration vers une plateforme offrant un contrôle total sur le robots.txt, car c'est un prérequis technique fondamental.
🏷 Related Topics
Domain Age & History Crawl & Indexing AI & SEO

🎥 From the same video 3

Other SEO insights extracted from this same Google Search Central video · duration 9 min · published on 21/10/2026

🎥 Watch the full video on YouTube →

Related statements

💬 Comments (0)

Be the first to comment.

2000 characters remaining
🔔

Get real-time analysis of the latest Google SEO declarations

Be the first to know every time a new official Google statement drops — with full expert analysis.

No spam. Unsubscribe in one click.