What happened with this product aggregator?

A site compiling thousands of automated product listings faced a shocking episode. Googlebot crawled 1.17 million URLs in a few weeks. Four pages indexed total. Even the homepage remained unindexed.

The gap between crawl and indexing indicates something simple: Google crawls a lot, indexes little as soon as it detects mass-produced content without qualitative filtering. Search Console graphs show this unambiguously, with a vertical crawl spike next to a desperately flat indexing curve, signature of massive algorithmic rejection.

What are the two technical problems identified?

The first issue: several URLs that are theoretically indexed redirect directly to the homepage. This symptom reveals a serious structural disorganization, likely due to poorly configured server rewrites, or chain redirects that have never been cleaned up.

The second problem is that the volume of automatically produced pages far exceeds what Google considers useful to index. Mueller puts it plainly: Google generally does not appreciate aggregation practices or programmatic SEO pushed to the extreme. This position is nothing new. It is rarely stated so bluntly.

Why does Google reject programmatic SEO at this scale?

The engine tolerates automatically generated sites if they provide value: price comparison sites fueled with fresh data, enriched listings with verified reviews, useful synthetic content. Duplicating 180,000 nearly identical pages to cover the long tail, however, is viewed as sophisticated spam.

To the naked eye, Google struggles to distinguish useful aggregation from parasitic aggregation. Its algorithms quickly spot template content patterns: minimal original text, repetitive structures, low user engagement, no external quality signals. Crawling continues to monitor, but indexing shuts down.

• Massive crawling never guarantees indexing: Googlebot can crawl millions of URLs without keeping a single one indexed if quality is lacking.
• Broken redirects are a critical warning sign: they indicate a poorly maintained site and lead Google towards cautious de-indexing.
• The volume of automated pages must be justified by real utility: does each URL meet a distinct search intent with differentiated content?
• Aggregation is not forbidden but closely scrutinized: Google applies strict quality filters to these sites and tolerates few technical errors.

Is this statement consistent with field observations?

Absolutely. For years, product aggregation sites have faced increasing indexing difficulties, a phenomenon exacerbated after the Helpful Content Update. Google has hardened its stance on mass-generated content, even if technically sound.

The ratio is surprising: 1.17 million crawls for 4 indexations. Two hypotheses. Either the site has accumulated critical technical errors that Mueller does not mention, catastrophic response times, massive internal duplicate content, circular canonicals. Or Google categorized the entire domain as low-quality from its initial analyses. [To be verified]: nothing indicates if any manual or algorithmic penalties were active.

What nuances should be added regarding programmatic SEO?

Not all programmatic sites suffer this fate. Amazon, Leboncoin, or Booking automatically generate millions of pages and remain perfectly indexed. The difference? Fresh user-generated content, rich structured data, strong engagement signals, and high domain authority.

An aggregator survives if it provides editorial or functional value: written comparative analyses, relevant advanced filters, exclusive data like historical prices or real-time availability. Google is not anti-programmatic; it is anti-useless. What condemns the analyzed site is not the automatic generation, but that the algorithm perceives no differentiation.

In what cases does this rule not apply?

A programmatic site accumulating natural editorial backlinks, media mentions, and organic social shares benefits from significantly greater tolerance. Domain authority weighs heavily. An established site can afford template content that a new domain would immediately be refused.

Ultra-specialized sites, with exhaustive coverage of a niche, form the second exception. A comparator dedicated solely to vintage car parts, with 50,000 detailed references and technical diagrams, can succeed where a generalist aggregator mixing household appliances, fashion, and high-tech would fail.

What should be prioritized in an audit of an aggregation site?

Start with the crawl/indexing ratio. In Search Console, a huge gap (greater than 100:1) almost always indicates a spiral of algorithmic rejection, and this is the first number to put on the table before any further checks. Then move on to redirects: export all URLs indexed according to Google, test them one by one, isolating those that redirect to the homepage without justification.

Next comes the question of similarity of your templates. Pull 100 pages at random and measure the percentage of unique content on each, excluding header, footer, and sidebar. Below 30% of truly different text, Google likely sees masked duplicates. Text fingerprinting tools can help you spot identical repeated blocks.

How can you reduce the volume of pages without losing existing traffic?

Identify pages that accumulate zero organic clicks over 12 months and have no backlinks. Remove them. If they serve internal navigation, a noindex is sufficient. Redirects should target relevant parent pages, never the homepage in bulk.

The editorial effort should focus on the pages that are already ranking: original content, manually written comparisons, exclusive data tables. Better to have 5,000 very rich pages than 180,000 skeletal listings. Google will index more once each page justifies its existence with perceived value.

What mistakes must absolutely be avoided with programmatic content?

No page should be generated for a combination lacking real search volume. "Red shoes size 47 delivery Cantal" likely has no monthly demand and dilutes your crawl budget for no reason. Filter viable combinations from keyword research data.

The same logic applies to empty or nearly empty pages left online while waiting for future content. Google crawls them, sees they provide nothing, and deteriorates the overall domain trust. Product unavailable? Either remove the page or offer relevant alternatives, never a blank template with just a title.

• Export all URLs indexed via Search Console and verify that none redirect to the homepage unjustifiably
• Measure the real unique content rate (excluding template) on a sample of 100 automatically generated pages
• Identify and remove/noindex all pages without organic traffic or backlinks in 12 rolling months
• Filter parameter combinations without documented search volume before generating new pages
• Manually enrich the 20% of pages that generate 80% of traffic with original editorial content
• Audit server response times: a slow aggregator drastically increases indexing problems