event archival: move old events to cold storage to bound store growth #148

New issue

Open

opened 2026-03-01 11:45:07 +00:00 by ash · 0 comments

ash commented

2026-03-01 11:45:07 +00:00

Owner

Problem

Event stores grow unbounded. A stream with 100K events means every Load reads 100K rows (before snapshot optimization). Even with snapshots, the data consumes disk and slows backups.

Solution

Event archival strategy:

Stream compaction — after snapshot at version N, archive events [1, N-buffer] to a separate table or external storage. Keep only recent events + snapshot.
Cold storage — move archived events to events_archive table (same schema, different table with less aggressive indexing) or export to S3/file.
Transparent loading — LoadFrom checks archive if main table doesnt have old events. Or just fail fast — if you need old events, restore from archive explicitly.

Scope

Archive events older than N versions behind snapshot
events_archive table (same DB, less indexing)
CLI command to trigger archival
Configurable retention policy per stream or globally
Both SQLite and Postgres

Priority

Medium — needed for long-running production systems, not MVP

## Problem Event stores grow unbounded. A stream with 100K events means every Load reads 100K rows (before snapshot optimization). Even with snapshots, the data consumes disk and slows backups. ## Solution Event archival strategy: 1. **Stream compaction** — after snapshot at version N, archive events [1, N-buffer] to a separate table or external storage. Keep only recent events + snapshot. 2. **Cold storage** — move archived events to `events_archive` table (same schema, different table with less aggressive indexing) or export to S3/file. 3. **Transparent loading** — LoadFrom checks archive if main table doesnt have old events. Or just fail fast — if you need old events, restore from archive explicitly. ## Scope - Archive events older than N versions behind snapshot - `events_archive` table (same DB, less indexing) - CLI command to trigger archival - Configurable retention policy per stream or globally - Both SQLite and Postgres ## Priority Medium — needed for long-running production systems, not MVP