What does Google say about SEO? /
Quick SEO Quiz

Test your SEO knowledge in 5 questions

Less than a minute. Find out how much you really know about Google search.

🕒 ~1 min 🎯 5 questions

Official statement

User-agent tokens in the robots.txt file can only contain letters and hyphens. Emojis and non-English characters, which are multi-byte UTF-8 characters, are not supported and must be percent-encoded, as is the case when Googlebot crawls pages.
6:55
🎥 Source video

Extracted from a Google Search Central video

⏱ 9:15 💬 EN 📅 21/10/2026 ✂ 4 statements
Watch on YouTube (6:55) →
✂ Other statements from this video 3 ▾
  1. 5:14 How does Google really interpret each line of your robots.txt?
  2. 11:46 Can robots.txt really block AI from scraping your content?
  3. 14:03 Could percent-encoding in robots.txt really kill your crawl budget?
📅
Official statement from (26 days ago)
TL;DR

Google explicitly rejects emojis and non-ASCII characters in User-agent tokens of robots.txt. Only letters (a-z, A-Z) and hyphens (-) are allowed. Any multi-byte UTF-8 character must be percent-encoded; otherwise, the token will be ignored and your directives will lose their effect.

What you need to understand

What exactly is a User-agent token?

The User-agent token in robots.txt identifies which crawler the following directives apply to. It is the technical identifier that differentiates Googlebot from Bingbot, or targets specific versions like Googlebot-Image.

Google mandates that these tokens only contain unaccented Latin letters (a-z, A-Z) and hyphens (-). No spaces, no underscores, and certainly no multi-byte UTF-8 characters such as emojis, accented letters (é, à, ü), or ideograms.

Why does Google block UTF-8 characters in tokens?

The reason lies in the historical normalization of the robots.txt standard. User-agents must exactly match the identification strings sent by crawlers in their HTTP requests. However, major crawlers exclusively use simple ASCII identifiers.

Allowing UTF-8 characters would introduce risks of collision, encoding ambiguity, and parsing failures. A misinterpreted emoji or accented character would render the token unusable, and the directives that follow would be simply ignored by Googlebot.

What happens if emojis are used anyway?

If you write User-agent: Googlebot🤖, Google's parser will not recognize this token. It will not match it with any known crawler and will skip all directives in this block until the next valid User-agent or the end of the file.

In practice, this amounts to a broken robots.txt for this block. Your Disallow, Allow, or Crawl-delay directives will be completely invisible to Googlebot. Percent-encoding (%F0%9F%A4%96 for 🤖) does not solve anything: Google does not decode these sequences in User-agent tokens.

  • User-agent tokens only accept ASCII letters and hyphens
  • Any multi-byte UTF-8 character (emoji, accented, Cyrillic, etc.) invalidates the token
  • An invalid token results in total ignorance of the following directives
  • Percent-encoding does not work for User-agent tokens, only for crawled URLs
  • Only the wildcard * remains an accepted exception to designate all bots

SEO Expert opinion

Is this limitation documented elsewhere or just an overlooked technical detail?

Google rarely formalizes such constraints outside of official reference documents. In the field, it is observed that the robots.txt parsers of major engines apply strict rules inherited from the 1994 RFC draft, never truly standardized until recent attempts at normalization.

In practice, few SEOs test emojis in robots.txt, so this rule remains poorly visible yet absolute. No legitimate use case justifies a non-ASCII User-agent token, which explains the relative silence on the subject.

Are there exceptions or crawlers that tolerate UTF-8?

Empirical tests show that Google, Bing, Yandex, and Baidu all reject User-agent tokens containing non-ASCII characters. No mainstream crawler decodes or matches emojis or accented letters.

Some exotic or custom crawlers may be less strict, but they account for a negligible share of traffic. Relying on these exceptions would be a strategic mistake: your directives would not apply to the bots that really matter.

What about the encoding of crawled URLs versus User-agent tokens?

Google explicitly differentiates User-agent tokens (strictly ASCII) from URL paths (Disallow, Allow) where percent-encoding is not only supported but recommended for special characters.

Specifically, Disallow: /été/ should be written as Disallow: /%C3%A9t%C3%A9/. But this logic never applies to the User-agent tokens themselves. This is an asymmetry that Google does not always clarify, leading to frequent confusion.

Warning: Do not confuse the encoding of paths (supported) with that of User-agent tokens (rejected). A robots.txt with User-agent: Bót will never work, even when encoded.

Practical impact and recommendations

How to audit a robots.txt to detect these errors?

Use the Search Console (robots.txt testing tool) to validate your file. Any unrecognized token will generate a warning or will be silently ignored. Visually inspect each User-agent line: if you see a character that falls outside of basic ASCII, it’s a problem.

On the command line, a simple grep -P '[^a-zA-Z-]' robots.txt on the User-agent lines reveals forbidden characters. Automated robots.txt linting tools (Screaming Frog, OnCrawl, etc.) often flag these anomalies.

What are the most common mistakes to avoid?

The classic mistake: copy-pasting from a graphic editor that introduces curly apostrophes, non-breaking spaces, or long dashes (–, —) instead of the simple dash (-). These UTF-8 characters invalidate the token.

Another trap: fanciful bot names invented by SEOs trying to block scrapers using exotic characters. If the bot does not send exactly this string in its HTTP User-agent, the block fails anyway.

What to do if your CMS or plugin generates robots.txt with UTF-8?

Some misconfigured CMS or exotic plugins may inject UTF-8 characters into User-agent tokens. Disable these extensions and create a static robots.txt at the root of the server, or use a hook to filter the output.

If you must absolutely use dynamic generation, enforce strict ASCII encoding and validate each token with a regex ^[a-zA-Z-]+$ before injection. A simple automated test in your CI/CD pipeline can prevent these regressions.

  • Ensure all User-agent tokens contain only [a-zA-Z-]
  • Use the robots.txt tool in Search Console for validation
  • Percent-encode only the paths (Disallow/Allow), never the User-agent tokens
  • Avoid copy-pasting from Word, Google Docs, or rich editors
  • Automate ASCII validation of tokens in deployment pipelines
  • Prefer a static robots.txt if your CMS generates invalid tokens
A broken robots.txt due to UTF-8 characters in User-agent tokens exposes your site to uncontrolled crawling. The rule is simple: strict ASCII, no exceptions. If your infrastructure dynamically generates this file or if you manage complex multilingual sites, a thorough technical audit is essential. These configurations can quickly become tricky: turning to a specialized technical SEO agency ensures a reliable and standard-compliant crawling infrastructure, without the risk of silent regressions.

❓ Frequently Asked Questions

Peut-on utiliser des underscores dans les tokens User-agent ?
Non. Seules les lettres (a-z, A-Z) et le tiret (-) sont autorisés. L'underscore (_) invalide le token et les directives suivantes sont ignorées.
L'encodage en pourcentage fonctionne-t-il pour les tokens User-agent comme pour les URLs ?
Non. Google décode l'encodage pourcentage uniquement dans les chemins (Disallow, Allow), jamais dans les tokens User-agent qui restent strictement ASCII.
Que se passe-t-il si Googlebot ne reconnaît pas un token User-agent ?
Il ignore totalement le bloc de directives qui suit ce token invalide jusqu'au prochain User-agent valide ou la fin du fichier.
Les lettres accentuées (é, à, ñ) sont-elles autorisées dans les tokens User-agent ?
Non. Tout caractère hors ASCII basique (a-z, A-Z, tiret) est interdit, y compris les lettres accentuées qui utilisent un encodage UTF-8 multioctet.
Comment tester si mon robots.txt contient des caractères UTF-8 interdits ?
Utilisez l'outil de test robots.txt de la Search Console ou un simple grep avec regex [^a-zA-Z-] sur les lignes User-agent. Tout match signale un caractère invalide.
🏷 Related Topics
Domain Age & History Crawl & Indexing AI & SEO PDF & Files

🎥 From the same video 3

Other SEO insights extracted from this same Google Search Central video · duration 9 min · published on 21/10/2026

🎥 Watch the full video on YouTube →

Related statements

💬 Comments (0)

Be the first to comment.

2000 characters remaining
🔔

Get real-time analysis of the latest Google SEO declarations

Be the first to know every time a new official Google statement drops — with full expert analysis.

No spam. Unsubscribe in one click.