We built a vector database engine in Rust to answer one question — how far can commodity hardware go before you actually need a managed-cloud bill? Not as a thought experiment: as a working single-node engine, currently in development, targeting 50M+ vectors, built under the same architect-led Method behind everything else we ship. Building a vector database from the storage layer up teaches you things no pricing page will — where memory actually goes, what durability actually costs, and which of the many tunable knobs genuinely matter. Here are the five lessons we'd hand to any team betting on AI retrieval infrastructure.
What did we build?
The engine is a single-node vector database written in Rust. Storage is an LSM tree — writes land in a memtable and flush into immutable segments, and each segment carries its own HNSW graph for approximate nearest-neighbour search. Vectors can be stored at full precision or quantized with SQ8 or product quantization (PQ). Distance kernels are hand-vectorized with AVX2 and AVX-512 SIMD. Every write passes through a CRC32C-checksummed write-ahead log before it is acknowledged. Interfaces are REST over axum and gRPC over tonic. Nothing exotic — just the pieces that matter, built so they could be measured.
Lesson 1 — Quantization is the real capacity lever
At float32, a 768-dimensional embedding costs 3 KB before any index overhead. Multiply by tens of millions of vectors and RAM — not compute — becomes the constraint that prices your hardware. Quantization changes that arithmetic: SQ8 stores each dimension in a single byte, and PQ compresses further still, the two together spanning roughly 4–32× memory savings depending on configuration. The price is recall, because quantized distances are approximations — and the only honest way to set the trade is to measure it. Hold out real queries, compute exact ground truth with brute-force search, and track recall@k before and after. Picking a quantization level off a blog post is how retrieval quality quietly degrades in production.
Lesson 2 — HNSW parameters are a cost dial, not a constant
HNSW gives you three main knobs, and they bill you differently. M — the graph's connectivity — and efConstruction are paid at build time: more memory per vector, longer indexing runs. efSearch is paid per query: a wider search beam buys recall at the cost of latency. Treat the trio as a dial rather than a constant. An offline enrichment job can afford generous, high-recall settings; an interactive search box cannot. Per-segment graphs help here too — index build cost is paid incrementally as segments flush and merge, instead of as one monolithic rebuild that holds the whole service hostage.
Lesson 3 — Durability is where a single-node engine earns trust
A distributed database can hide sloppy persistence behind replication. A single-node engine cannot — the write-ahead log is the whole story. Ours checksums every record with CRC32C before the write is acknowledged, so a crash mid-write becomes a detectable, recoverable event at replay rather than silent corruption discovered weeks later. The other half of the storage story is compaction. LSM trees defer work: segments accumulate, and a background process merges them. Schedule that process badly and it competes with queries for CPU and I/O at exactly the wrong moments; let it fall behind and read amplification climbs instead. In practice, your p99 latency is your compaction-scheduling policy.
Lesson 4 — SIMD pays for itself, portability costs engineering time
Distance computation dominates query CPU, and hand-vectorized kernels earn their keep — the same dot product over 768 dimensions is dramatically cheaper when the loop processes eight or sixteen floats per instruction. We ship AVX2 as the baseline and use AVX-512 where the silicon offers it, selected by runtime dispatch. The honest footnote: every kernel variant is code you now own. Two instruction sets means two implementations to test, benchmark, and keep numerically consistent — worth it for an engine whose entire job is throughput, but it's engineering time a team should budget for up front rather than discover in month three.
Lesson 5 — Know your numbers before renting someone else's margin
Alongside the engine, we produced a full 3-site datacenter design study — engineered for 1M users at 9 ms latency, with a complete capex/opex model down to power and cooling. Not because every team needs a datacenter; almost none do. The point of the exercise is the model itself. When you know what serving your workload costs on hardware you specify, a managed-cloud quote stops being a mystery and becomes a comparison. Sometimes the managed service wins on those numbers, and you buy it with confidence. What you never have to do again is rent someone else's margin blind.
Why does a boutique studio build infrastructure at all?
This is R&D, and it feeds client work directly. AI features stand on data infrastructure — retrieval, embeddings, latency budgets — and a team that has built that layer from the metal up estimates it honestly instead of optimistically. It's the same reasoning behind our AI delivery model: use the new tools aggressively, but keep people who understand the layer beneath them. The engine lives in our AI & data infrastructure portfolio line, marked In Development — because that's what it is, and statuses should mean something.
Planning retrieval or AI features and unsure what the infrastructure should honestly cost? We'll walk you through the real numbers before you commit to anyone's bill.