How the storage engine actually works — the read and write paths, the durability argument and its limits, the sharding and replication model, the search subsystem, and how the engine handles failure. Written for the engineer who'll be on call for this in production.
A first-principles walk through the storage layer, the I/O model, the sharding scheme, and exactly when an acknowledged write is durable on commodity NVMe — and when it is not.
VeltrixDB is open source under Apache 2.0 — every line is on GitHub. But reading the Go source isn't how most teams evaluate a database, so we also wrote this architectural document. It's the fastest way for a CTO or security-review team to understand exactly how the engine works — and the source is right there when they want to go deeper.
How VeltrixDB separates keys from values — the 64 B index entry (~142 B/key of RAM in practice, plus ~90 B/key for the ordered index), the VLog record format (4 KB-aligned, magic + CRC32C), block packing (~26 × 128 B records per 4 KB block, on the MultiPut and GC paths only — a single Put takes a whole block), and why write amplification is ~1× by design (not measured) when there is no LSM compaction.
Reads are one Go pread per cache miss; a write batch is one pwrite and one fdatasync. Why the io_uring write bridge is opt-in (off by default), and what it measured on a 4-CPU CI runner.
Adaptive group commit (default since v1.12: a lone writer is synced at once; 15 ms is only the upper bound), why VLog and WAL fdatasync run concurrently, full replay after a crash, the compacted checkpoint written on clean shutdown, Raft vs replicated mode (one or the other, never both), and the conditions under which an acknowledged write can be lost.
One FNV-1a hash drives three placements: 8192 index shards, up to 256 cache shards, and the disk. Across nodes, a consistent-hash ring (64 virtual nodes each) that the client routes on — the server does not forward — over static --peers membership, and --rack-id placement that keeps copies of a partition in different racks or zones.
The LIRS algorithm in detail, value-aware eviction (values ≤ 256 B get priority 2 and are kept longer; a 16-entry window from the cold end evicts the largest, lowest-priority value first), and why the cache is split into up to 256 shards (one lock was 21.7% of CPU; a hit went from 711 ns to ~92 ns).
Crash restart, silent disk corruption, a slow disk, GC falling behind, node loss. For each: what the engine does, which metric shows it, and what is tested today (end-to-end failover, persistence and backup/restore suites).
HNSW vector indexes (float32, int8 or product-quantized; layer-0 graph on the heap or in a mapped file), a BM25 inverted index, and reciprocal-rank fusion of the two — all derived from ordinary durable keys and filtered by the KV record with the same id. Shipped in v1.11 and v1.12.
TTLs, INCR / CAS / SETNX, hashes with per-field TTL, lists, sets, namespaces, ordered RANGE / SCANCUR, secondary indexes, and one-shot TXN (per-key version check, read-committed, not MVCC). No sorted sets, pub/sub, streams, Lua or RESP.
Two pulled-out figures from the document — the write path, and the failure matrix.
A PUT is compressed, then encrypted (if enabled), then appended to the VLog and the WAL concurrently. The caller waits for max(WAL, VLog), not their sum. Concurrent writers share a single write(2) and a single fdatasync; only then is the index entry installed, the cache updated and the write acknowledged. Since v1.12 group commit is adaptive by default: a lone writer is synced at once, concurrent writers after an idle gap of about one fdatasync, and the 15 ms window is only the upper bound (--group-commit=fixed restores the full wait). With an emulated 300 µs fdatasync on macOS, a lone writer's P50 went from 16 ms to 0.45 ms and 64 writers from 4.0K to 37.5K writes/s.
The chapter covers the binary WAL record (48-byte header, CRC32C over the whole record; the default since v1.6, with --wal-format=text as rollback), crash replay, the compacted checkpoint written on clean shutdown, and the exact conditions under which an acknowledged write survives a single-node crash.
The failure chapter maps each failure to what the engine does about it and the Prometheus metric that shows it happening. It also says what is tested today — the end-to-end suites cover node failover, persistence across restarts and backup/restore — and what is not.
The point of publishing this isn't to say nothing ever fails — it's to say which failures we know about, and how you'll see them. We don't publish recovery-time numbers we haven't measured reproducibly.
Every section, every figure number, every page. If a chapter is what you're after, we'll send you the chapter — you don't have to take the whole PDF.
--peers membership, rack-aware placement, rebalancingp. 26A taste of the failure chapter — what the engine does in each case and the Prometheus metric that shows it. Where something is covered by the end-to-end test suites, we say so; we don't quote recovery times we haven't measured reproducibly.
A crash (or kill -9) means a full replay of wal.log on start. Writes are acknowledged only after WAL and VLog fdatasync, so on Linux an acknowledged write is on disk (macOS fsync returns at the drive cache: crash-safe, not power-loss-safe). A clean shutdown instead writes a compacted checkpoint (one record per live key) and renames it over the WAL. Covered by the persistence e2e suite.
The legacy text WAL truncated replay for keys containing | or a newline, so acknowledged writes could be lost on a crash restart. The binary WAL (default since v1.6) fixes this and reads old text records mixed in.
Every VLog record carries a magic number and a CRC32C, checked on every read. A background scrubber walks the log; any mismatch increments veltrixdb_scrub_corruption_total. Any non-zero rate means investigate.
Per-disk read and write latency EWMAs are sampled continuously. A disk at ≥ 5× the median raises veltrixdb_vlog_disk_slow. Each disk has its own WAL and VLog, so a failure on one disk does not touch the others.
GC pauses under read pressure, which alone could stall it forever. It escalates in three tiers and at 65% garbage bypasses the pause unconditionally; a pause left over from a write-only period also clears once no read has arrived for 4 minutes. Each bypass increments veltrixdb_vlog_gc_emergency_runs_total; a persistent rate means writes exceed sustainable GC bandwidth.
A node is FAILED after 10 s without a heartbeat and is left out of search fan-out, but it stays on the hash ring, so the client keeps routing its keys to it. In --mode=raft a lost leader is replaced by election (400–800 ms timeout) and writes follow MOVED. In --mode=replicated on v1.13 and earlier a replica whose send fails is marked FAILED and is not brought back until the primary restarts. Fixed in v1.14: failed replicas are re-probed with backoff and caught up (SYNC_PENDING → SYNC), and a RECOVERING node returns to ACTIVE after 2 good pings. On v1.14 and earlier a node marked FAILED also starts auto-rebalance, which deletes keys outside the Raft log: killing the leader of a loaded 3-node raft cluster left the survivors with 5,435 and 7,875 of 20,000 keys. Run with --auto-rebalance=false until you upgrade. Fixed in v1.15: node failures never migrate keys, raft mode never migrates, and replicated mode deletes a key only after every replica acknowledged it. Nothing re-creates lost copies. Covered by the node-failover e2e suite; we don't publish recovery times we haven't measured reproducibly.
Chapter 9 documents the engine changes that shipped from v1.6 to v1.12. Every number states its conditions, including the ones that did not go our way.
Each shard’s index moves into a C++ table off the Go heap on cgo builds (macOS included). Go’s GC no longer scans it. Measured at 5M keys: full GC 21 → 0.27 ms, settled RSS 168 → 142 B/key, for ~19 ns more per cache-miss lookup. Needs vm.max_map_count ≥ 262144 at large key counts.
48-byte little-endian header, CRC32C over the whole record. Fixes lost writes on crash replay for keys with | or a newline. Encoding 4× faster per record, replay 2.5× faster. --wal-format=text writes the legacy format as a rollback.
Overwriting a live key no longer re-inserts it into the RANGE skiplist. Server batch writes (8 clients × 1024-key MPUT, 1M keys, macOS): 1.45M → 3.54M keys/s, P99 9.6 → 4.2 ms.
--net=cpp: an event loop per core, binary protocol only, standalone mode only, opt-in. On a 4-CPU CI runner: read P50 154 vs 286 µs, but read P99 ~3.0 vs 1.8 ms and batch writes ~12% lower. Not faster overall. Disk-bound reads now finish off the loop (deferred reads); not re-measured yet.
It used to start on every Linux cgo build. A VLog batch is already one pwrite + one fdatasync, and with the C++ storage extras all on the CI runner measured 1.12M vs 1.77M keys/s batch writes. Per-part attribution is pending; scripts/net-bench.sh bisects it. VELTRIXDB_URING_BRIDGE defaults to off.
The WAL and VLog flushers share one pacer. A lone writer is synced at once; concurrent writers after an idle gap of about one fdatasync (EWMA, clamped 20 µs–2 ms); the 15 ms window is only the upper bound. Emulated 300 µs sync, macOS: lone-writer P50 16 → 0.45 ms; 64 writers 4.0K → 37.5K writes/s at the same writes per fdatasync. Durability is unchanged.
A freshly elected leader used to serve a local GET before applying everything its predecessor had acknowledged. Reads now wait (≤ 2 s) for the leader’s no-op to apply. Without it a unit test missed the acknowledged write in 20 of 20 failovers; with it, 0 of 20. In v1.12 it covers GET and MGET only: hash, namespace and RANGE / SCANCUR reads stay local. Fixed in v1.14: every read on both protocols passes it (namespace, hash, list, set, range, VER, IDXQUERY, QUERY and searches).
Nightly: a 20-minute search soak checked against an oracle of acknowledged writes, and 10 SIGKILL / restart cycles. In 3 local cycles every acknowledged vector, document and record was present after restart and no acknowledged delete came back. Writes in flight at the kill are not checked.
cpp/ also holds an ART index, a priority io_uring scheduler, an SQPOLL reader, a C++ VLog and an eBPF GC throttle. They compile, and nothing in Go calls them. The read path is Go pread. The chapter says so explicitly rather than listing them as features.
Since v1.11 the engine serves nearest-neighbour vector search, BM25 full-text search and hybrid search next to the records they describe. The full command reference, tuning and every measurement are in the vector & hybrid search docs.
PUT doc-42, VSET doc-42 and TSET doc-42 describe one thing. Vectors persist as @vec/<ns>/<id> (float32), documents as @txt/<ns>/<id>, settings as @vecns/<ns> — ordinary durable keys. The RAM indexes are derived state, updated by engine hooks on Put / Delete / MultiPut, so replication, Raft apply and restore index correctly. Similarity is cosine.
Full float32 vectors always stay in the VLog; RAM holds per namespace either float32, int8 codes or PQ codes, with the HNSW layer-0 edges optionally in a mapped file (GRAPH disk). Measured heap per 768-dim vector (10K synthetic, macOS): 3,383 B float32 · 1,078 B int8 · 493 B PQ · 378 B PQ + disk graph (+216 B mapped). int8 / PQ re-rank against the float32 vectors with one parallel MultiGet.
Okapi BM25 (k1 = 1.2, b = 0.75) over a Unicode tokenizer — no stemming, stop words or phrase queries. HSEARCH fuses the vector and text rankings by weighted Reciprocal Rank Fusion, α/(60 + rank), α = 0.5 by default; ranks are fused, so cosine and BM25 never need a common scale.
FILTER field op value is evaluated on the KV record whose key is the vector id. An = filter on a field with a secondary index (IDXCREATE) is served from the index — up to 2,048 matches are scored exactly without a graph walk; other filters are checked during a capped walk.
Search writes go through the normal Raft or replicated path, and the ring places a record’s vector, text and index entries on the record’s node. Every search runs on all non-failed nodes and is merged (BM25 with cluster-wide statistics). A peer that misses --search-timeout-ms fails the request unless --search-allow-partial. Node-to-node traffic is signed with HMAC-SHA256 (--cluster-secret-file) or mTLS.
The graph and inverted index are not saved: they are rebuilt from the keys at every start, and searches are refused until that finishes (INFO → search_ready). GloVe-100, 100K real vectors, float32, ef = 256: recall@10 0.953, p50 0.77 ms (one query at a time, macOS). Tested to 100K real vectors; nothing at 10M+, and no same-hardware comparison with another vector database.
One email gets you the PDF — including the chapter you skipped. If you want a specific chapter, mention it in the body and we'll send just that.