Official statement
Other statements from this video 3 ▾
Google emphasizes strict adherence to the robots.txt standard, particularly the use of percent-encoding for special characters. Improperly encoded robots.txt can lead to directives being ignored by some crawlers, exposing critical sections or incorrectly blocking strategic resources. The stakes go beyond mere formal compliance: it's about effective control of crawling.
What you need to understand
What is percent-encoding in robots.txt?
Percent-encoding involves representing special characters by their hexadecimal code, preceded by the % sign. For instance, a space becomes %20, and an acute accent %C3%A9. This standard, defined in RFC 3986, ensures that URLs containing non-ASCII characters are interpreted uniformly by all user agents.
In robots.txt, this rule applies to path patterns declared after Disallow or Allow. If your directory contains URLs with accents, spaces, or Cyrillic characters, you must encode those portions. Otherwise, the crawler may interpret the pattern differently or even ignore it entirely.
Why is Google reinforcing this technical point now?
This statement likely comes after field observations of poorly formed robots.txt files. Many CMS generate this file automatically without strictly adhering to the RFC, especially for multilingual sites or legacy databases with non-normalized URLs.
The issue arises when a webmaster manually adds a directive for a path containing a special character. For example: Disallow: /données/ instead of Disallow: /donn%C3%A9es/. The crawler may then fail to recognize the pattern and crawl the section that was supposed to be blocked, wasting crawl budget or exposing unfinished content.
Do all crawlers uniformly adhere to this standard?
No, and this is precisely the crux of the message. Googlebot has strictly applied the standard for years, but other bots (Bing, third-party crawlers, scrapers) have variable implementations. Some tolerate raw characters, while others silently ignore them.
The result: a functional robots.txt for Googlebot may be bypassed by Bingbot or an AI crawler. This is why there is insistence on a firm adherence to the standard: it's the only guarantee for predictable multi-crawler interpretation.
- Always encode non-ASCII characters in robots.txt path patterns
- Strict adherence to RFC 3986 is the only guarantee of uniform interpretation by all crawlers
- A poorly encoded pattern may be silently ignored, exposing critical sections or incorrectly blocking resources
- Tolerance varies between crawlers: what works for Googlebot may fail elsewhere
SEO Expert opinion
Is this directive applied uniformly by Google itself?
Googlebot has indeed strictly adhered to the RFC 3986 standard since at least 2019. Tests show that patterns containing unencoded raw characters are treated literally: if the actual path is encoded in the URL, the raw pattern does not match. Conversely, an encoded pattern correctly matches the actual encoded path.
However, Google has long tolerated certain deviations in Search Console, displaying warnings rather than blocking errors. This statement marks a shift in tone: the advice is becoming almost mandatory to ensure expected behavior.
What are the real risks of a poorly encoded robots.txt?
The first risk is unintentional exposure of sensitive content. A real-life example: an e-commerce site blocked /panier/ with a raw pattern containing a tilde ~, which was misinterpreted by some crawlers. Result: tens of thousands of cart pages indexed, generating duplication and diluting crawl budget.
The second risk is accidental blocking of critical resources. A pattern that is too broad or poorly formed can prevent crawling of strategic categories. The last underestimated risk is incompatibility with new AI crawlers (ChatGPT, Perplexity) that apply the standard to the letter. [To be verified]: no public data on their tolerance rate for encoding errors.
In what cases can this rule be disregarded?
If your structure is 100% ASCII (URLs in English, without accents or special characters), the question does not arise. But as soon as a site involves multilingual aspects, legacy URLs with spaces, or complex GET parameters, encoding becomes essential.
Be cautious of false positives: some automated audits report encoding errors when the robots.txt is correct. Always manually check with the Test robots.txt file tool in Search Console, which strictly applies Google's parser.
Practical impact and recommendations
How can I check if my robots.txt respects the encoding standard?
The first step: run the file through the Search Console robots.txt tester. Manually enter URLs containing special characters from your directory and check that the behavior (allowed/blocked) corresponds to your intention.
The second method: use an online RFC 3986 validator to identify non-ASCII characters in your patterns. Any character outside the range [a-zA-Z0-9-._~:/?#[]@!$&'()*+,;=] must be encoded. A simple Python script with urllib.parse.quote() can automate the conversion.
What should I do if my CMS generates a non-compliant robots.txt?
Many CMS (WordPress, Drupal, Magento) dynamically generate robots.txt through hooks. If you detect an issue, switch to static mode: create a robots.txt file at the root of the server and disable dynamic generation. This gives you total control over the format.
For SaaS platforms where you lack direct file access (Shopify, certain builders), check for manual editing options for robots.txt. If no option exists, contact technical support with a clear reference to RFC 3986. A non-compliant robots.txt is a legitimate bug.
What common mistakes should I absolutely avoid?
The number one mistake: copy-pasting URL paths from a browser without checking the actual encoding. The browser often displays decoded characters for readability, but the underlying technical URL is encoded. Always compare with the raw address bar or the network inspector.
The second trap: using wildcards (*) without testing their scope with encoded characters. A pattern /cat*/ may not match /cat%C3%A9gorie/ depending on the implementation. Always test each critical pattern with real URLs from your site.
- Validate the robots.txt in Search Console with real URLs containing special characters
- Consistently encode any non-ASCII character with percent-encoding (e.g., é → %C3%A9)
- Switch to static mode if the CMS generates a non-compliant robots.txt
- Test patterns with wildcards on encoded URLs to verify matching
- Automate the verification with a script validating RFC 3986
- Document each directive for easier maintenance
❓ Frequently Asked Questions
L'encodage percent-encoding est-il obligatoire pour tous les caractères spéciaux dans robots.txt ?
Un robots.txt mal encodé peut-il pénaliser mon référencement ?
Comment tester rapidement si mon robots.txt respecte le standard RFC 3986 ?
Les crawlers d'IA (ChatGPT, Perplexity) respectent-ils ce standard d'encodage ?
Que faire si mon CMS ne permet pas d'éditer manuellement le robots.txt ?
🎥 From the same video 3
Other SEO insights extracted from this same Google Search Central video · duration 9 min · published on 21/10/2026
🎥 Watch the full video on YouTube →
💬 Comments (0)
Be the first to comment.