How does Google truly identify duplicate pages on your site?

What does Google mean by 'content without boilerplate'?

Google does not blindly compare the raw HTML of two pages to decide if they are duplicates. The engine first extracts the unique main content — what Google calls the 'main content' — by discarding everything that repeats across multiple pages: navigation, sidebars, footers, recurring headers.

Specifically, if your template includes 600 words of boilerplate and only 150 words change between two pages, Google only looks at those 150 words. If those 150 words are identical or nearly identical across two different URLs, you have a duplicate in the eyes of the algorithm — even if the visual appearance is different.

What are the issues caused by soft 404 errors?

A soft 404 error is a page that no longer exists or contains nothing useful but returns a 200 HTTP code ('all is well') instead of a 404 or 410. Typically: empty category pages, search results with no results, deleted product sheets still accessible with a generic message.

The trap: all these pages often display the same error message or the same empty template. Google extracts the main content, finds 50 identical words across 20 different URLs, and concludes that there is a cluster of duplicates. The result: these pages are likely to be grouped together, and only one version will be indexed — often not the one you would have chosen.

What is the reasoning behind this approach?

Google aims to save its crawl budget and not pollute its index with redundant content. If the engine detects that 80% of your pages share the same main content, it will naturally reduce the crawl frequency and limit the number of indexed pages.

This reasoning is based on the assumption that duplication is rarely intentional and signals an architecture problem. By ignoring boilerplate, Google tries to focus on the real added value of each page — but this approach assumes that your HTTP codes are correct and that your templates are well thought out.

• Google compares the main content, not the full HTML or visual layout.
• Boilerplate is ignored: menus, footers, repeated elements do not count in duplicate detection.
• Soft 404 errors create false duplicates if they share the same generic message across multiple URLs.
• Returning correct HTTP codes (404, 410, 301) is essential to prevent Google from confusing errors with legitimate content.
• The architecture of your templates directly influences Google's ability to distinguish your pages.

Is this statement consistent with real-world observations?

Yes, but with important nuances. Tests indeed show that Google ignores repeated blocks when comparing content. However, the boundary between 'boilerplate' and 'main content' is not always clear to the algorithm.

We regularly observe cases where information-rich blocks — recurring FAQs, standard comparison tables, partially templated product descriptions — are treated as boilerplate because they repeat across multiple pages. [To be verified]: Google does not publish any precise threshold for what shifts from 'useful content' to 'ignored boilerplate'. The opacity remains total on the exact criteria.

What concrete risks are there if your HTTP codes are misconfigured?

An e-commerce site with 500 permanently out-of-stock product sheets, all returning a 200 code with the message 'This product is no longer available', ends up with 500 almost identical pages in the eyes of Google. The engine will then group these URLs, index only a handful, and drastically reduce the crawl of other sections.

Worse still: if these soft 404 pages receive backlinks or are included in your XML sitemap, Google will allocate budget to crawl them regularly — even though they shouldn't exist. You waste your resources on dead URLs, and your true strategic pages may be explored less frequently.

In what cases does this rule become a trap?

Sites with significant pagination or dynamically generated content are particularly exposed. Imagine a listings site: each results page shows 10 listings + 300 words of boilerplate (filters, generic SEO text). If a search returns no results, the page displays 'No listings found' — and Google sees almost no main content.

The result: hundreds of empty results pages are treated as duplicates of each other, even if the URL parameters differ. If you do not block these pages (robots.txt, noindex, 404 code), your site becomes a crawl budget sinkhole.

What immediate steps should be taken to avoid false duplicates?

First, audit your HTTP codes. Any page that no longer has a reason to exist should return a 404 or 410, not a 200. Use a crawler like Screaming Frog or OnCrawl to identify pages with little unique content and check their response codes.

Next, ensure that the unique content / boilerplate ratio is favorable on each type of page. If your template includes 500 words of repeated text and your product sheets only add 100 unique words, Google may group them. Turn the tide: reduce boilerplate or enhance the specific content.

What mistakes should absolutely be avoided?

Never leave empty or nearly empty pages accessible with a 200 code. This is the typical case of category pages without products, author pages without articles, orphan tag pages. Either remove them (404/410), redirect them (301), or block them upfront (noindex, robots.txt).

Avoid also multiplying identical generic error messages. If you need to display a 'product unavailable' page, customize the content with alternative suggestions, similar products, or redirect directly to a relevant category. The worst scenario: 200 pages displaying 'Sorry, this page does not exist' with zero variation.

How to check if your site complies?

Crawl your site and filter pages by content similarity rate. Most SEO tools (Oncrawl, Sitebulb, Botify) offer this feature. Identify clusters of pages with nearly identical content and verify their legitimacy.

Then, cross-reference with Google Search Console reports: 'Coverage' section, 'Excluded' tab. If you see 'Detected, currently not indexed' or 'Crawled, currently not indexed' on strategic pages, it often signifies that Google is treating them as duplicates or low-value pages.

• Configure appropriate HTTP codes: 404 for deleted pages, 410 for permanent deletions, 301 for redirects.
• Reduce boilerplate in your templates or enhance the unique content for each page.
• Block or delete empty pages, search results with no results, or pagination without content.
• Customize error messages to avoid creating clusters of identical pages.
• Crawl your site regularly to detect low unique content pages.
• Monitor Search Console reports for mistakenly excluded pages.