Does your BigQuery Search Console data really pile up forever without limits?

Why doesn't Google handle expiration automatically?

The bulk export of Search Console to BigQuery is designed as a one-way data flow: Google pushes the data, that's it. Managing the data lifecycle — how long to keep this data, when to archive it or delete it — remains entirely your responsibility.

This aligns with BigQuery's logic: it's your data warehouse, not a service managed by Google Search Console. You decide what you do with it, but also what it costs you.

What actually happens if you don't do anything?

Data accumulates month after month. Each day brings its load of new rows: queries, impressions, clicks, average positions. On a site with significant volume, this can represent millions of rows monthly.

Without an expiration policy, you end up with a dataset that grows indefinitely. And growing storage means growing costs — BigQuery charges based on storage volume and the volume scanned during queries.

What are your options for controlling retention?

BigQuery offers several mechanisms: table-level expiration, dataset-level expiration, or conditional deletion scripts via SQL. You can also use partitioned tables with automatic expiration of older partitions.

• Set a default expiration date on the dataset (e.g., 13 months)
• Define specific expiration per table if you want more granular control
• Use time-based partitions with automatic expiration of partitions exceeding a threshold
• Implement an automated script that deletes data beyond a certain age

Is this logic really justified on Google's side?

Let's be honest: from Google's perspective, it makes sense. BigQuery isn't a Search Console product, it's a general-purpose data warehousing service. Google can't guess how long you want to keep your data — some analysts want 3 years of history, others purge after 6 months.

Moreover, imposing a rigid default expiration would be frustrating: imagine losing 2 years of history because an automatic rule deleted your tables without warning. So yes, you need to manage it. But — and here's where it gets sticky — Google makes no effort to warn naive users. No alerts in the interface, no reminders during initial setup. Result: many discover the problem after receiving a hefty invoice. [To verify] that a notification is added in Search Console during first-time setup.

In what cases does this accumulation become problematic?

On a small site with 10k monthly impressions, the financial impact remains marginal even after 2 years without cleanup. But on a site with several million daily impressions — e-commerce, media, marketplace — data explodes quickly.

The real risk, beyond storage costs, is the query cost. If your dashboards systematically scan 3 years of data when you're only analyzing the last 6 months, you're burning budget unnecessarily. That's where a smart partitioning strategy becomes essential.

What's the right retention duration for a typical SEO site?

It depends. For standard SEO analysis, 13 months is often sufficient: it covers a full year plus 1 month of rolling comparison. If you're working on multi-year seasonality analysis or longitudinal studies, 24-36 months can be justified.

Beyond that, usefulness drops rapidly. Search Console data older than 3 years is rarely actionable — the competitive landscape, the algorithm, your site have changed too much. Better to export annual aggregates to another tool if you want to keep lightweight historical traces.

How do you configure expiration on BigQuery in practice?

Multiple levels of action possible. The simplest: set a default expiration at the dataset level. It applies to all tables it contains, present and future.

You can do this via the BigQuery interface ("Details" tab of the dataset, "Default table expiration" section) or via the bq CLI: bq update --default_table_expiration 31536000 my_dataset (duration in seconds, here 365 days).

For a more granular approach, use date-partitioned tables with automatic partition expiration. Example: one partition per day, expiration after 395 days. This lets you maintain precise history while automatically purging the old.

What mistakes should you absolutely avoid?

• Don't set expiration at all — the default trap, the one 80% of users fall into
• Configure expiration too short and lose strategic data mid-analysis
• Forget to test the configuration on a dev dataset before applying it to production
• Fail to document the retention policy with your team — 6 months later, no one knows why data is disappearing
• Systematically scan all data without filtering by partition — query costs explode

What should you monitor once the config is in place?

Regularly verify that expiration works as expected. Quick check in BigQuery: are old tables disappearing properly? If you see tables from 2 years ago when your policy specifies 13 months, something's wrong.

Also monitor your BigQuery costs via Google Cloud Console. If storage keeps climbing despite configured expiration, you may have a configuration issue or tables outside the main dataset that escape the rule.