feat: LiveProjection — subscribable projections with fan-out for SSE #47

New issue

Closed

opened 2026-02-20 10:46:44 +00:00 by ash · 1 comment

ash commented

2026-02-20 10:46:44 +00:00

Owner

Problem

1000 users watching same dashboard should NOT mean 1000 store subscriptions.

Design

LiveProjection[E, V] — one EventSubscription consumer, in-memory view, fan-out to N clients via Go channels.

type LiveProjection[E any, V any] struct {
    view        V
    subscribers map[string]chan V
    evolve      func(V, eskit.Event[E]) V
}

One consumer against the event store (EventSubscription)
In-memory read model updated by the single consumer
Fan-out to N SSE clients via Go channels (~96 bytes per client)
Zero I/O per client after initial load
Slow client handling — non-blocking send, drop if buffer full

SSE Integration

SSE handler subscribes to projections, not the store:

orders := orderProj.Subscribe(ctx)
revenue := revenueProj.Subscribe(ctx)
// Multiplex into one SSE stream

One SSE connection per client, multiple projections, real-time.

Scaling

1000 clients same dashboard = 1 store subscription + 1000 channels
Fan-out is a for-loop, microseconds
Memory: ~100KB for 1000 subscribers
No database/NATS pressure per client

Tests

1 event → N subscribers receive it
Slow subscriber → update dropped, not blocking
Subscribe/unsubscribe lifecycle
Concurrent updates + subscribes
1000 subscriber fan-out benchmark
Memory profile: 10k subscribers
Integration with Datastar SSE

## Problem 1000 users watching same dashboard should NOT mean 1000 store subscriptions. ## Design `LiveProjection[E, V]` — one EventSubscription consumer, in-memory view, fan-out to N clients via Go channels. ```go type LiveProjection[E any, V any] struct { view V subscribers map[string]chan V evolve func(V, eskit.Event[E]) V } ``` - **One consumer** against the event store (EventSubscription) - **In-memory read model** updated by the single consumer - **Fan-out** to N SSE clients via Go channels (~96 bytes per client) - **Zero I/O per client** after initial load - **Slow client handling** — non-blocking send, drop if buffer full ## SSE Integration SSE handler subscribes to projections, not the store: ```go orders := orderProj.Subscribe(ctx) revenue := revenueProj.Subscribe(ctx) // Multiplex into one SSE stream ``` One SSE connection per client, multiple projections, real-time. ## Scaling - 1000 clients same dashboard = 1 store subscription + 1000 channels - Fan-out is a for-loop, microseconds - Memory: ~100KB for 1000 subscribers - No database/NATS pressure per client ## Tests - 1 event → N subscribers receive it - Slow subscriber → update dropped, not blocking - Subscribe/unsubscribe lifecycle - Concurrent updates + subscribes - 1000 subscriber fan-out benchmark - Memory profile: 10k subscribers - Integration with Datastar SSE

ash added the

enhancement

performance

labels

2026-02-20 10:46:44 +00:00

ash commented

2026-02-20 10:57:53 +00:00

Author

Owner

Update: Topic-based subscriptions

Subscribers must be able to filter by key — order ID, tenant ID, user ID, etc.

orders.Subscribe(ctx, "order-123")       // specific order
orders.Subscribe(ctx, "tenant:acme")     // tenant filter
orders.Subscribe(ctx, "*")              // wildcard (admin)

Routing keys extracted from events via user-defined function:

LiveProjection{
    RoutingKeys: func(event Event[E]) []string {
        return []string{event.StreamID, "tenant:" + event.TenantID}
    },
}

Fan-out only to matching subscribers. 1000 users watching different orders = 1000 keyed subs, each getting only their updates. Zero wasted bandwidth.

## Update: Topic-based subscriptions Subscribers must be able to filter by key — order ID, tenant ID, user ID, etc. ```go orders.Subscribe(ctx, "order-123") // specific order orders.Subscribe(ctx, "tenant:acme") // tenant filter orders.Subscribe(ctx, "*") // wildcard (admin) ``` Routing keys extracted from events via user-defined function: ```go LiveProjection{ RoutingKeys: func(event Event[E]) []string { return []string{event.StreamID, "tenant:" + event.TenantID} }, } ``` Fan-out only to matching subscribers. 1000 users watching different orders = 1000 keyed subs, each getting only their updates. Zero wasted bandwidth.

ash referenced this issue from a commit

2026-02-20 12:03:14 +00:00

feat: add LiveProjection with topic-based fan-out + SSE helper, full E2E integration test

ash closed this issue

2026-02-20 12:03:14 +00:00