Catch-up subscriptions — resume from position, then stream live #58

New issue

Closed

opened 2026-02-21 22:42:50 +00:00 by ash · 0 comments

ash commented

2026-02-21 22:42:50 +00:00

Owner

Problem

Projections currently rebuild from scratch on startup. No way to say "give me all events from position X, then keep streaming new ones."

Proposal

type Subscription[E any] interface {
    Subscribe(ctx context.Context, opts SubscribeOptions) (<-chan Event[E], error)
}

type SubscribeOptions struct {
    FromPosition   uint64    // global sequence to resume from
    EventTypes     []string  // filter (reuses #54)
    BatchSize      int       // how many to fetch per poll
}

Phases:

Catch-up: load historical events from FromPosition in batches
Live: switch to real-time notifications (SQLiteNotifier / NATS / pg LISTEN)
Seamless: no gap between catch-up and live

Why This Matters

Projections checkpoint their position and resume without full rebuild
Enables horizontal scaling (competing consumers each track their position)
Foundation for automations/reactors that survive restarts

Acceptance

SubscribeOptions with FromPosition, EventTypes, BatchSize
SQLite implementation using existing SQLiteNotifier
Checkpoint storage (where does the projection save its position?)
Integration test: write events during catch-up, verify no gaps
Graceful shutdown (drain in-flight, save checkpoint)

## Problem Projections currently rebuild from scratch on startup. No way to say "give me all events from position X, then keep streaming new ones." ## Proposal ```go type Subscription[E any] interface { Subscribe(ctx context.Context, opts SubscribeOptions) (<-chan Event[E], error) } type SubscribeOptions struct { FromPosition uint64 // global sequence to resume from EventTypes []string // filter (reuses #54) BatchSize int // how many to fetch per poll } ``` Phases: 1. **Catch-up**: load historical events from `FromPosition` in batches 2. **Live**: switch to real-time notifications (SQLiteNotifier / NATS / pg LISTEN) 3. **Seamless**: no gap between catch-up and live ## Why This Matters - Projections checkpoint their position and resume without full rebuild - Enables horizontal scaling (competing consumers each track their position) - Foundation for automations/reactors that survive restarts ## Acceptance - [ ] SubscribeOptions with FromPosition, EventTypes, BatchSize - [ ] SQLite implementation using existing SQLiteNotifier - [ ] Checkpoint storage (where does the projection save its position?) - [ ] Integration test: write events during catch-up, verify no gaps - [ ] Graceful shutdown (drain in-flight, save checkpoint)