Official statement
The robots.txt file uses exact URLs. If you have entries that refer to a specific version of a URL, they will not apply to other versions of that URL.
Other statements from this video 6 ▾
- □ Why are URLs case-sensitive, and why does it matter for SEO?
- □ Should you worry about URL variations when it comes to crawling?
- □ How does duplicate URL canonicalization affect your SEO?
- □ How can internal linking shape your SEO strategy?
- □ How does the canonical tag really impact your SEO?
- □ Should you really worry about case sensitivity in URLs for SEO?
Official statement from
(5 years ago)
⚠ A more recent statement exists on this topic
Should you really block AI crawlers in your robots.txt?
View statement →
TL;DR
The robots.txt file uses **exact URLs**. This means that an entry for a specific version of a URL will not apply to other versions. Be sure to check each variant of the URL.
❓ Frequently Asked Questions
Que se passe-t-il si une URL est majuscule ?
Si les URL majuscules ne sont pas precisement indiquees, elles ne seront pas interpretees par le fichier robots.txt.
Le fichier robots.txt doit-il etre modifie regulierement ?
Oui, il doit etre revu lors de changements structurels pour assurer que les nouvelles URLs sont correctement incluses.
Les caracteres speciaux sont-ils pris en compte ?
Les caracteres speciaux sont distingues et traites precisement, comme toute URL differente.
🎥 From the same video 6
Other SEO insights extracted from this same Google Search Central video · published on 28/09/2021
🎥 Watch the full video on YouTube →
💬 Comments (0)
Be the first to comment.