Official statement
<p>Robots.txt prevents the crawling of a URL but does not stop its indexing. Google may index a URL blocked by robots.txt based solely on its URL and the link anchors pointing to it. To prevent indexing, use the noindex meta tag.</p>
Other statements from this video 24 ▾
- □ How does Google identify the abusive use of expired domains?
- □ Should You Tackle DMCA Complaints One URL at a Time for Better SEO?
- □ Does temporary deindexing really impact your SEO ranking?
- □ Why should you care about content copies that don’t affect your SEO ranking?
- □ Do omitted results really impact your SEO?
- □ Does the way you spell really influence your SEO ranking?
- □ Why does brand traffic remain resilient against algorithm updates?
- □ Why do certain brands fail to show up for generic search queries?
- □ How can you stand out in SEO within a saturated market?
- □ How does Google really determine the publication dates of a page?
- □ Why should very large sites rethink their use of Search Console?
- □ Why should you leverage the Search Console API for large SEO sites?
- □ Should core updates be considered in isolation for SEO?
- □ Does the number of reviews truly impact your SEO?
- □ What makes your pages remain invisible on Google?
- □ Is it true that your site can't be manually reassessed for quality?
- □ Does AMP really enhance your site's speed without hurting SEO performance?
- □ Do affiliate links really impact your SEO?
- □ Why doesn't Google penalize every single SEO mistake?
- □ Why doesn’t removing a redirect boost your SEO?
- □ Should you unlock JavaScript and CSS for Google?
- □ How does changing the URL of an optimized image hasten its update in the index?
- □ Are Core Web Vitals really based on actual user data?
- □ Does the choice between WordPress pages and posts really impact SEO?
Official statement from
(5 years ago)
⚠ A more recent statement exists on this topic
Should you really block AI crawlers in your robots.txt?
View statement →
TL;DR
Google can index a URL even if it is blocked by robots.txt. Only the noindex meta tag prevents indexing. Be cautious of link anchors that may reveal a URL.
❓ Frequently Asked Questions
Comment bloquer efficacement une URL de l'indexation ?
Utilisez la balise meta noindex. Le robots.txt ne suffit pas pour cela.
Quels risques si seules les instructions de robots.txt sont utilisées ?
L'URL peut malgré tout être indexée si elle est référencée par des liens externes.
La balise noindex suffit-elle à bloquer le crawl ?
Non, elle empêche l'indexation mais pas le crawl, contrairement au robots.txt.
🎥 From the same video 24
Other SEO insights extracted from this same Google Search Central video · published on 06/08/2021
🎥 Watch the full video on YouTube →
💬 Comments (0)
Be the first to comment.