Vector Quantization

Vector quantization is a family of compression techniques that shrink embeddings so a vector index fits in less memory and searches faster. A raw embedding might be 1,536 float32 numbers — about 6 KB each; at ten million vectors that's 60 GB of RAM. Quantization approximates each vector with a much smaller code: scalar quantization drops the precision (float32 to int8), while product quantization splits the vector into chunks and replaces each with the ID of a nearby centroid. The trade-off is accuracy for cost. Compressed vectors return slightly less precise nearest-neighbor results, but often close enough — and you can re-rank the top candidates against full-precision vectors to recover most of the loss. For SaaS builders running RAG or semantic search at scale, quantization is the difference between an index that fits in RAM and one that doesn't. Practical note: measure recall against your own queries before and after — the acceptable trade-off is workload-specific, not universal.