Official statement
Crawling is blocked using the robots.txt file. Random users on the web can still access the website while well-behaved search engines will know not to access it.
Other statements from this video 6 ▾
- □ How can you conceal your site from Google search results while keeping it accessible?
- □ Is using a password the most effective way to safeguard private content?
- □ Is it true that robots.txt doesn't guarantee your URLs won't be indexed?
- □ How does the noindex tag influence your SEO indexing strategy?
- □ How does password protection impact the SEO of private content?
- □ When is it a good idea to block crawling or indexing for your SEO optimization?
Official statement from
(4 years ago)
⚠ A more recent statement exists on this topic
Should You Really Block the GoogleOther Crawler in Your Robots.txt?
View statement →
TL;DR
The robots.txt file prevents disciplined search engines from crawling, but it doesn't restrict access for users or less cooperative tools. Implication: Ensure that sensitive content is protected by other means if necessary.
🎥 From the same video 6
Other SEO insights extracted from this same Google Search Central video · published on 24/11/2021
🎥 Watch the full video on YouTube →
💬 Comments (0)
Be the first to comment.