Store migration tool (migrate between backends) #116

New issue

Closed

opened 2026-02-23 09:30:56 +00:00 by ash · 0 comments

ash commented

2026-02-23 09:30:56 +00:00

Owner

Summary

CLI tool to migrate event data between any two eskit-supported stores (SQLite ↔ Postgres ↔ NATS).

Nobody in the Go ES space has this. Users start with SQLite, grow into Postgres, maybe NATS. Right now they'd write custom scripts. We make it one command.

Usage

eskit migrate --from sqlite://app.db --to postgres://prod-host/db
eskit migrate --from postgres://old/db --to nats://cluster:4222 --batch-size 1000
eskit migrate --from sqlite://app.db --to sqlite://backup.db --dry-run

Features

Reads all streams from source, writes to target
Preserves: stream IDs, event order, metadata, timestamps, correlation IDs
Batch processing with configurable batch size
Resumable via checkpointing (track last migrated global position)
Dry-run mode (validate without writing)
Progress reporting (stream count, event count, throughput)
Codec conversion support (e.g., migrate from JSON to CBOR simultaneously)
Verification pass: compare source vs target event counts and checksums

Implementation

Both stores implement EventStore[any] + GlobalReader — so it's "read all from source, append to target" with batching and checkpointing. The interfaces already support this cleanly.

Testing Requirements

Full round-trip tests: SQLite→Postgres, Postgres→SQLite, SQLite→NATS, etc. (all permutations)
Data integrity verification: checksums match after migration
Resumability test: interrupt mid-migration, resume, verify completeness
Dry-run test: verify zero writes to target
Large dataset test: 100K+ events migration
Codec conversion test: migrate JSON store to CBOR store, verify decode
Edge cases: empty store, single event, stream with 1M events
Concurrent migration safety (no corruption under -race)

Benchmark Requirements

Benchmark migration throughput (events/sec) per store combination
Benchmark with varying batch sizes (10, 100, 1000, 10000)
Benchmark codec conversion overhead during migration
Memory profiling: ensure streaming (not loading all events into memory)
Compare with/without verification pass overhead

Part of CLI

This will be a subcommand of the eskit CLI (#115).

## Summary CLI tool to migrate event data between any two eskit-supported stores (SQLite ↔ Postgres ↔ NATS). Nobody in the Go ES space has this. Users start with SQLite, grow into Postgres, maybe NATS. Right now they'd write custom scripts. We make it one command. ## Usage ``` eskit migrate --from sqlite://app.db --to postgres://prod-host/db eskit migrate --from postgres://old/db --to nats://cluster:4222 --batch-size 1000 eskit migrate --from sqlite://app.db --to sqlite://backup.db --dry-run ``` ## Features - Reads all streams from source, writes to target - Preserves: stream IDs, event order, metadata, timestamps, correlation IDs - Batch processing with configurable batch size - Resumable via checkpointing (track last migrated global position) - Dry-run mode (validate without writing) - Progress reporting (stream count, event count, throughput) - Codec conversion support (e.g., migrate from JSON to CBOR simultaneously) - Verification pass: compare source vs target event counts and checksums ## Implementation Both stores implement `EventStore[any]` + `GlobalReader` — so it's "read all from source, append to target" with batching and checkpointing. The interfaces already support this cleanly. ## Testing Requirements - Full round-trip tests: SQLite→Postgres, Postgres→SQLite, SQLite→NATS, etc. (all permutations) - Data integrity verification: checksums match after migration - Resumability test: interrupt mid-migration, resume, verify completeness - Dry-run test: verify zero writes to target - Large dataset test: 100K+ events migration - Codec conversion test: migrate JSON store to CBOR store, verify decode - Edge cases: empty store, single event, stream with 1M events - Concurrent migration safety (no corruption under -race) ## Benchmark Requirements - Benchmark migration throughput (events/sec) per store combination - Benchmark with varying batch sizes (10, 100, 1000, 10000) - Benchmark codec conversion overhead during migration - Memory profiling: ensure streaming (not loading all events into memory) - Compare with/without verification pass overhead ## Part of CLI This will be a subcommand of the eskit CLI (#115).