42 pages of how the storage engine actually works — block-by-block diagrams, the durability proof, the sharding model, and the failure modes we test in chaos engineering. Written for the engineer who'll be on call for this in production.
A first-principles walk through the storage layer, the I/O model, the sharding scheme, and the proofs behind a 99.999% durability SLA on commodity NVMe.
VeltrixDB is open source under Apache 2.0 — every line is on GitHub. But reading 40k lines of Go isn't how most teams evaluate a database, so we also shipped this 42-page architectural document. It's the fastest way for a CTO or security-review team to understand exactly how the engine works — and the source is right there when they want to go deeper.
How VeltrixDB separates keys from values — index layout, VLog segment format, pointer encoding, and why compaction that never touches values is the only way to guarantee 1.0× write amplification at scale.
How we structure the submission and completion queues per shard, why we use O_DIRECT, and the benchmarks comparing io_uring vs pread at our P99 targets.
Group-commit WAL, replication invariants, the actual probability calculus behind the published durability number, and the conditions under which it does not hold.
Why 1024, how we rebalance without read interruption, the operator's reshard procedure, and the cap on individual shard ownership during partition events.
The LIRS algorithm in detail, the priority-2 eviction-resistance tweak for small hot keys, and the measured hit-rate uplift on Zipfian workloads vs LRU.
The full failure-injection matrix — single-node loss, full-zone outage, NVMe bit-flip, kernel hang, network partition. Every test case, the expected behaviour, and the RTO/RPO numbers we hit.
Two pulled-out figures from the document — the write path, and the failure-injection matrix. The full PDF has 18 diagrams of this density.
The five-stage write pipeline. Application code lands in a per-shard ring buffer; a dedicated commit thread wakes every 50µs (or on batch-full), issues a single fsync, and unparks every blocked writer at once. The 80µs fsync cost is divided across the entire batch — at 2M ops/sec, that's roughly ~0.85µs per write for full durability.
The full chapter covers the lock-free queue design, the futex-park behaviour, the WAL crash-recovery protocol, and a proof that the WAL never loses an acknowledged write under any single-node failure.
The chaos-engineering matrix lives in chapter 7. We run each of these injections nightly against a production-equivalent cluster. The chapter documents the exact fault-injection spec, the expected user-observable behaviour, and the recovery time we measured over a 90-day window.
The point of publishing this isn't to say nothing ever fails — it's to say we know what does, and we test for it. When you have an incident, the failure mode you'll hit is almost certainly already in the matrix.
Every section, every figure number, every page. If a chapter is what you're after, we'll send you the chapter — you don't have to take the whole PDF.
A taste of what's in the chaos chapter — every one of these is exercised against a production-equivalent cluster every night, and the results are published to the same Prometheus we expose on customer clusters.
We kill -9 a healthy node at random under sustained 2M ops/sec load. Expected: reads continue from replica within 8 seconds, no acknowledged writes lost. Measured RTO: 8.4s P99.
iptables-drop all traffic from a zone for 30 minutes. Expected: traffic shifts to remaining zones, write quorum maintained on 2/3 zones. Measured RTO: 22s · zero data loss.
Inject a deliberate bit-flip into a sealed VLog segment. Expected: checksum trips on next read, value re-fetched from replica, bad segment quarantined. Detection < 1s.
Stall a CPU running the io_uring SQPOLL thread for 30s. Expected: liveness probe trips, the node is auto-ejected from the read pool, traffic re-routes to healthy nodes. Ejection in 12s.
Cut QUIC replication between two regions for 5 minutes. Expected: both regions continue serving local reads & writes, conflict resolution applies on heal. RPO < 30s, no acknowledged writes lost.
Force every shard into aggressive GC simultaneously while serving peak load. Expected: GC respects its cgroup weight, P99 reads stay under 8ms. Measured P99 during storm: 6.4ms.
Chapter 9 documents five features that shipped together in the v0.9.5 build cycle — each one targeting a specific latency or cost failure mode we observed in production. This is a sample of that chapter.
VeltrixDB's I/O layer switched to polling mode — the engine monitors the completion queue continuously rather than waiting for kernel interrupts. Under sustained write load, this eliminates wakeup overhead entirely. Measured: 0 syscalls per read op.
The GC engine now ranks VLog entries by access temperature. Hot data is never relocated during a GC pass. Cold entries compact first. Result: GC runs without competing for the NVMe bandwidth live reads need. GC emergency runs: 0 in 60-min stress.
The compaction thread is pinned to a reserved CPU core on Linux. It never contends with request-handling threads for scheduler time. The intermittent P99 spikes under heavy write pressure — a known failure mode on shared-core designs — are eliminated. P99 jitter: ±3% under load.
A built-in BPF program watches CPU throttle counters in real time and signals the GC scheduler before a throttle event reaches the write path. Hard stalls become gradual backpressure. 0 hard stalls in 60-min sustained write stress.
VeltrixDB's tiering engine moves data inactive for >5 minutes to S3, GCS, or Azure at 20 MB/s, rate-limited. The NVMe index stays hot. The GC engine is aware of tiered entries and never relocates them. Cold storage cost: 10× lower than NVMe.
All five improvements together, verified against a 200K-key sustained write benchmark: 110k ops/s sustained · P99 162ms per batch · density 160 B/record · 0 GC emergency runs. Both packing density and GC gates pass.
One email gets you the 42-page PDF — including the chapter you skipped. If you want a specific chapter, mention it in the body and we'll send just that.