Codec system: registry, protobuf, buffer pooling, transport encoding #22

New issue

Closed

opened 2026-02-19 21:59:52 +00:00 by ash · 1 comment

ash commented

2026-02-19 21:59:52 +00:00

Owner

What

Upgrade eskit serialization to a full codec system. Reference: Myrra codec package (YoYoPass pkg/myrra/codec).

Current State

eskit has Serializer interface (JSON, jsoniter, Gob) + EventRegistry for type resolution. Works but missing key features.

Needed

1. Codec Registry

Named codecs registered by name ("json", "jsoniter", "protobuf", "gob")
Thread-safe registry with RWMutex
Select codec per event type or globally
Store codec name with each persisted event so decode always works

2. Protobuf Codec

First-class proto.Message support
5-10x faster than JSON for large payloads
Use for performance-sensitive event types

3. Buffer Pooling

sync.Pool for JSON encoder buffers (Myrra does this, eskit does not)
Max pool buffer size cap (64KB) to prevent memory waste
Initial buffer size 512 bytes
Free perf win, especially for high-throughput appends

4. Transport Encoding (Event Codec)

EncodeEvent(event, codec) → Message (data + headers)
DecodeEvent(message, registry) → Event
Headers carry: event ID, type, timestamp, version, schema version, stream, codec name, causation, correlation, principal, custom metadata
All headers use es- prefix convention
es-meta- prefix for custom metadata
This is how events travel over NATS, HTTP, or any transport

5. Per-Event Codec Selection

Store es-codec header with each event
Allows mixing JSON and protobuf in same store
Critical for migration: change default codec without breaking existing events

Integration

EventStore.Append should accept codec option
EventStore.Read should use stored codec name to decode
CommandBus should propagate codec preference

Benchmarks

JSON vs jsoniter vs protobuf vs gob (encode + decode)
With and without buffer pooling
Various event sizes (100B, 1KB, 10KB, 100KB)
Save as BENCHMARK.txt baseline

TigerStyle

Max codec name length
Max registered codecs
Bounds on message/header sizes
nil checks on all inputs

## What Upgrade eskit serialization to a full codec system. Reference: Myrra codec package (YoYoPass pkg/myrra/codec). ## Current State eskit has `Serializer` interface (JSON, jsoniter, Gob) + `EventRegistry` for type resolution. Works but missing key features. ## Needed ### 1. Codec Registry - Named codecs registered by name ("json", "jsoniter", "protobuf", "gob") - Thread-safe registry with RWMutex - Select codec per event type or globally - Store codec name with each persisted event so decode always works ### 2. Protobuf Codec - First-class proto.Message support - 5-10x faster than JSON for large payloads - Use for performance-sensitive event types ### 3. Buffer Pooling - sync.Pool for JSON encoder buffers (Myrra does this, eskit does not) - Max pool buffer size cap (64KB) to prevent memory waste - Initial buffer size 512 bytes - Free perf win, especially for high-throughput appends ### 4. Transport Encoding (Event Codec) - `EncodeEvent(event, codec) → Message` (data + headers) - `DecodeEvent(message, registry) → Event` - Headers carry: event ID, type, timestamp, version, schema version, stream, codec name, causation, correlation, principal, custom metadata - All headers use `es-` prefix convention - `es-meta-` prefix for custom metadata - This is how events travel over NATS, HTTP, or any transport ### 5. Per-Event Codec Selection - Store `es-codec` header with each event - Allows mixing JSON and protobuf in same store - Critical for migration: change default codec without breaking existing events ## Integration - EventStore.Append should accept codec option - EventStore.Read should use stored codec name to decode - CommandBus should propagate codec preference ## Benchmarks - JSON vs jsoniter vs protobuf vs gob (encode + decode) - With and without buffer pooling - Various event sizes (100B, 1KB, 10KB, 100KB) - Save as BENCHMARK.txt baseline ## TigerStyle - Max codec name length - Max registered codecs - Bounds on message/header sizes - nil checks on all inputs