Gary Illyes

★★ Are you wasting your crawl budget on JavaScript files that add no value?

If a significant proportion of crawl budget (35% or 90%) is consumed by JavaScript files that add no content, it is recommended to consolidate these files or use X-Robots-Tag headers to prevent their ...

Aug 25, 2022 ⚡ Analysis available

★★★ Are 404s and robots.txt Really Wasting Your Crawl Budget?

HTTP status codes 404 and 410, as well as URLs blocked by robots.txt, do not consume crawl budget because Google only receives the status code without content. Conversely, soft 404s (pages that return...

Aug 25, 2022 ⚡ Analysis available

★★ Should you block your decorative JavaScript files to optimize your crawl budget?

If JavaScript files are purely decorative and add neither content nor value to the page rendering, they can be blocked via robots.txt or X-Robots-Tag. Rendering will fail for these resources but this ...

Aug 25, 2022 ⚡ Analysis available

★★★ Why is Google now rejecting certain directives in your robots.txt file?

The robots.txt file should only be used to control crawling. Google removed certain directives like noindex from the robots.txt parser because they don't concern crawling. Indexing and serving must be...

Aug 04, 2022 ⚡ Analysis available

★★★ Can Google really index your URLs blocked by robots.txt without you being able to do anything about it?

If a page blocked by robots.txt disallow is very popular on the internet, Google can index the URL without the content. The result will appear without a snippet but with the title. In this case, a met...

Aug 04, 2022 ⚡ Analysis available

★★★ Does Google really process HTTP status codes during the crawl phase, not after?

404, 410, and 403 codes are detected during the crawling phase. Google sends a signal to the indexing system indicating that the URL no longer exists, and indexing then decides to remove it. The crawl...

Aug 04, 2022 ⚡ Analysis available

★★★ Does Google extract meta robots and canonical tags during indexing rather than at crawl time—and why does this distinction matter for your site?

Meta tags such as meta robots noindex and rel canonical are extracted during the indexing process, when Google parses the content and performs rendering. If Google detects meta robots noindex, the URL...

Aug 04, 2022 ⚡ Analysis available

★★ Can a single noindex tag on an hreflang page contaminate your entire international cluster?

In hreflang clusters, pages can influence each other mutually. A noindex on one page in a cluster can potentially affect the entire cluster in case of detected duplication. It is recommended to add no...

Aug 04, 2022 ⚡ Analysis available

★★ Should you really rely on JavaScript to handle noindex directives?

Adding meta noindex via JavaScript should work in most cases after rendering, but there can be time windows or situations where rendering doesn't happen and the directive wouldn't be seen. It's better...

Aug 04, 2022 ⚡ Analysis available

★★★ Should you use X-Robots-Tag to keep PDFs and binary files out of Google's index?

For binary files like PDFs where it's impossible to add a meta tag, Google supports the X-Robots-Tag HTTP header with noindex. This header works like a meta noindex and will be processed during indexa...

Aug 04, 2022 ⚡ Analysis available

★★ Does the unavailable_after directive really slow down Google's crawling?

The unavailable_after directive leads to a gradual slowdown in crawling over time. Google continues to check the page because it wants to ensure the truthfulness of the information, just as it does wi...

Aug 04, 2022 ⚡ Analysis available

★★ Should you disable Google cache to take control of how your snippets appear in search results?

At the serving level (search results display), it is possible to control the snippets displayed and the cached content of the page. The meta noarchive tag allows you to remove the cached page without ...

Aug 04, 2022 ⚡ Analysis available

★★★ Does Google's removal tool actually delete your URLs from the index?

Google's removal tool masks results at the serving level but doesn't remove the URL from the index. Removal is fast because it happens at search time. For complete index removal, you also need to add ...

Aug 04, 2022 ⚡ Analysis available

★★ Why does it really take Google months to permanently remove a page from its index?

In some cases, it can take several months to completely remove a result from Google's actual index. This is why the removal tool exists: it allows immediate masking at the serving level while permanen...

Aug 04, 2022 ⚡ Analysis available

★★★ Does Google's URL removal tool actually stop crawling your pages?

Using the removal tool doesn't signal Google to stop crawling the page. Google will continue its crawl attempts. To prevent the page from returning to the index, you must combine the tool with noindex...

Aug 04, 2022 ⚡ Analysis available

★★ Could Google start rationing your website's crawl budget for environmental reasons?

Google is increasingly thinking about sustainability and the environmental impact of its crawl and indexation processes, suggesting a potential future evolution in web exploration practices....

Jul 26, 2022 ⚡ Analysis available

★★ Does consistency really beat sporadic effort in SEO?

Consistency is highly important in SEO. It is recommended to maintain a regular and steady routine in content production and monitoring of SEO information sources, rather than trying to do everything ...

Jul 14, 2022 ⚡ Analysis available

★★★ Does robots.txt really prevent your pages from being indexed by Google?

The robots.txt file limits what crawlers can explore on a site, but does not block indexation. If a page becomes very popular with many links, Google can still index the URL without the content, displ...

Jun 30, 2022 ⚡ Analysis available

★★★ Is robots.txt really ineffective at preventing your pages from being indexed by Google?

To reliably prevent a page from being indexed in Google Search, you must use the meta robots 'noindex' tag rather than robots.txt, as the latter is not a foolproof method against indexation....

Jun 30, 2022 ⚡ Analysis available

★★ Can you block indexation of entire directories using server modules instead of robots.txt?

To block indexation of a large portion of a site, you can use Apache modules or Nginx configurations to automatically apply the noindex tag to all URLs under a given prefix or pattern, although this i...

Jun 30, 2022 ⚡ Analysis available