Reduce SQLite store load allocations (30 allocs/event → target <5) #56

New issue

Closed

opened 2026-02-21 22:42:29 +00:00 by ash · 0 comments

ash commented

2026-02-21 22:42:29 +00:00

Owner

Problem

Benchmarks show 299K allocs for loading 10K events (~30 per event). This is excessive.

Sources likely include:

JSON deserialization allocating per-event
Row scanning creating new strings per field
Slice growth without pre-allocation
Metadata map creation per event

Investigation

Profile allocation sources with go test -benchmem -memprofile
Identify top 3 allocation hotspots

Proposed Fixes

Pre-allocate result slice with estimated capacity
sync.Pool for reusable JSON decode buffers
Reuse sql.Rows scan destinations
Skip metadata map creation when empty (most events have no extra metadata)
Consider jsoniter as default deserializer (4x faster deserialize per benchmarks)

Acceptance

Profile identifies top allocation sources
Allocs per event reduced from ~30 to <5
Benchmark before/after with exact numbers
No functional changes

## Problem Benchmarks show 299K allocs for loading 10K events (~30 per event). This is excessive. Sources likely include: - JSON deserialization allocating per-event - Row scanning creating new strings per field - Slice growth without pre-allocation - Metadata map creation per event ## Investigation - Profile allocation sources with `go test -benchmem -memprofile` - Identify top 3 allocation hotspots ## Proposed Fixes - Pre-allocate result slice with estimated capacity - `sync.Pool` for reusable JSON decode buffers - Reuse `sql.Rows` scan destinations - Skip metadata map creation when empty (most events have no extra metadata) - Consider `jsoniter` as default deserializer (4x faster deserialize per benchmarks) ## Acceptance - [ ] Profile identifies top allocation sources - [ ] Allocs per event reduced from ~30 to <5 - [ ] Benchmark before/after with exact numbers - [ ] No functional changes