Official statement
When standardizing robots.txt, Google expanded the language to accept URIs (not just URLs) to avoid creating new crawling control mechanisms for every new type of URI that might emerge on the internet. This includes, for example, deep links from app indexing.
Other statements from this video 13 ▾
- □ Why should SEO professionals still have a grasp of HTML today?
- □ Why is HTML crucial for effective SEO?
- □ Why is mastering JavaScript becoming crucial for SEO professionals?
- □ Are URLs truly untouchable when it comes to SEO?
- □ Why is the rel=canonical tag still crucial for SEO today?
- □ Is it true that Google is holding back on new meta tags in SEO?
- □ How are structured data essential for SEO control?
- □ Will Structured Data Become Obsolete for SEO?
- □ Why is Google focusing on intelligent crawling instead of pushing?
- □ Are Links Really Still Essential for SEO?
- □ Should you be wary of machine-generated content to avoid SEO penalties?
- □ Why are alt attributes crucial for image SEO?
- □ Is it true that voice search is really going to transform SEO as we know it?
Official statement from
(4 years ago)
⚠ A more recent statement exists on this topic
Should you really block AI crawlers in your robots.txt?
View statement →
TL;DR
Google now uses URIs in robots.txt to avoid creating new mechanisms for each new type of URI. This directly impacts the management of resources to ignore in app indexing.
❓ Frequently Asked Questions
Pourquoi Google inclut-il les URI dans robots.txt ?
Pour éviter de créer des mécanismes distincts à chaque nouvelle apparition de type d'URI et renforcer la gestion globale des règles de crawl.
Les URI impactent-ils le référencement naturel ?
Potentiellement oui, car ils permettent une gestion plus fine du crawl budget et influence implicitement l'indexation générale.
Dois-je revoir mon fichier robots.txt avec cette mise à jour ?
Oui, il est recommandé de vérifier que les URI pertinents sont inclus ou exclus selon votre stratégie de contenu.
🎥 From the same video 13
Other SEO insights extracted from this same Google Search Central video · published on 11/11/2021
🎥 Watch the full video on YouTube →
💬 Comments (0)
Be the first to comment.