Official statement
A no-index tag on a page does not stop Googlebot from crawling the page; it only prevents its indexing. On the other hand, blocking via robots.txt truly prevents crawling. If you see Googlebot crawling a URL blocked by robots.txt, they are probably false Googlebots.
Other statements from this video 13 ▾
- □ How can you prevent the indexing failure of your web pages?
- □ Why do links without HREF hinder Google's indexing?
- □ Why does Google index blocked pages by robots.txt?
- □ How does Google view window.history.pushState in terms of SEO?
- □ Is it true that the mobile validator isn't always reliable for your URLs?
- □ Does mobile compatibility really have a significant impact on your SEO?
- □ How can CDNs disrupt your site's mobile display?
- □ Is it true that JavaScript-loaded content remains invisible to Googlebot?
- □ Could HTML Buttons Be Undermining Your SEO Efforts?
- □ How can you ensure that every URL in your SPA is properly indexed by Google?
- □ How Does the Compatibility of the Beacon API with the WRS Influence Your SEO Tracking?
- □ How can you ensure that content beneath the viewport is acknowledged by Google?
- □ Are Google’s Testing Tools Truly Reliable for SEO?
Official statement from
(5 years ago)
⚠ A more recent statement exists on this topic
Why do so many SEO professionals still confuse robots.txt and no-index? Here's w...
View statement →
TL;DR
The no-index tag prevents indexing but not crawling. To block crawling, use the robots.txt file. Be cautious of false Googlebots if you notice crawling despite a block.
❓ Frequently Asked Questions
Une balise no-index empêche-t-elle le crawl ?
Non, elle empêche seulement l'indexation, pas le crawl.
Comment vérifier si un Googlebot est légitime ?
Utilisez les outils de Google pour vérifier les IP et comparez-les aux listes officielles.
Que faire en cas de visites par des faux Googlebots ?
Surveillez vos journaux de serveur et adaptez vos protections contre les bots frauduleux.
🎥 From the same video 13
Other SEO insights extracted from this same Google Search Central video · published on 22/09/2021
🎥 Watch the full video on YouTube →
💬 Comments (0)
Be the first to comment.