Segmented storage, sparse indexing, ISR replication, epoch-fenced leader election, and group commit batching: sustaining 245k records/sec at P99 < 20ms.
Eight phases, each building on the last. Every layer is independently tested and documented.
Group commit amortises fsync cost across concurrent writers: one sync covers
hundreds of records instead of one. Batch size is emergent: more concurrent producers
means larger batches, which means fewer fsyncs per record.
ISR-based replication with high-watermark consistency ensures consumers never read un-replicated records. Epoch-fenced leader election prevents stale leaders from writing after a new leader has been elected: zero split-brain, zero acknowledged-record loss.
Deterministic simulator with seeded RNG: same seed → same fault schedule every run. Crashed leaders, network partitions, message delays and reorderings: all reproducible. Found 7 correctness bugs invisible to unit tests.
Built incrementally: each phase has its own PDF writeup, test suite, and can be read in isolation.
CRC-stamped records, 8-byte sparse index entries, segment rolling, crash recovery. 7 tests including property test and torn-write simulation.
Length-prefixed binary framing, semaphore-based backpressure, graceful SIGTERM shutdown, linger-timer producer batching. Fuzz-tested decoder.
Per-partition append-only logs, key-hashed routing, sticky producer assignment. Per-partition ordering guarantee.
Group coordinator, sticky partition assignment to minimise rebalance churn, durable offset commits to an internal log.
Leader–follower log replication, high-watermark tracking, follower fetch loop, ISR shrink and expand on lag detection.
Durable monotonic EpochStore, controller polling loop, NeedsElection / Elect pure functions, Fence() rejects stale epochs. 5 tests including split-brain scenario.
SimClock (zero wall-clock cost), seeded Network with drop/delay, invariant checker (CheckAllOwned, CheckMonotonic, CheckNoRegression), Reproduce() for replay. 10 tests, race-clean.
SequenceTracker dedup, EncodeKey wire format, appendNoSync + batch sync(), MaxBatchSize / MaxBatchWait config, CI perf gate with 10% regression threshold.
Pure Go: no external runtime dependencies in the broker binary.