Embedding Index

An embedding index is the internal data structure that a vector database or search engine constructs over a collection of embeddings so that finding the nearest neighbors to a query vector doesn't require comparing it against every stored vector one by one. Without an index, similarity search is "brute force" (technically called a flat or exhaustive search) — accurate, but its cost grows linearly with the number of vectors, which becomes unworkably slow once a collection reaches millions of entries. Why it matters for AI/SaaS builders: the index type and its configuration are usually the single biggest lever on a RAG or semantic-search feature's latency, recall (how often the true best matches are actually returned), and infrastructure cost — and it's a decision every team building on raw pgvector, Qdrant, or Weaviate has to make explicitly (managed services like Pinecone hide it behind sane defaults). How it works: the dominant index family in production today is HNSW (Hierarchical Navigable Small World) — a graph structure where each vector is a node connected to its approximate nearest neighbors across multiple layers, letting search "hop" toward the query vector in roughly logarithmic time instead of scanning everything. HNSW offers excellent recall/speed trade-offs but is memory-hungry (the whole graph typically needs to stay in RAM) and index builds are computationally expensive to update incrementally at very high write volume. IVFFlat (Inverted File with Flat compression) is a cheaper alternative used by pgvector and others: it clusters vectors into buckets (via k-means) at index-build time and only searches the buckets nearest the query, trading some recall for lower memory and faster builds — good for datasets that don't change constantly. Tuning parameters like `ef_construction`/`ef_search` (HNSW) or the number of `lists`/`probes` (IVFFlat) directly trade recall against latency: searching more of the graph or more clusters finds better matches but takes longer. Worked example: a code-search SaaS indexes 2 million function embeddings. With a flat (no-index) search, a single query takes ~800ms — unacceptable for an IDE plugin. Switching to an HNSW index with `m=16, ef_construction=200` drops query latency to ~8ms with better than 95% recall against the brute-force ground truth, at the cost of the index needing ~6GB of RAM instead of the raw vectors' ~4GB on disk. The team later raises `ef_search` from 50 to 100 after noticing recall dip on ambiguous queries, trading a couple of extra milliseconds of latency for measurably better answer quality in the downstream LLM — a tuning trade-off that's invisible to the product surface but directly shapes whether "search for functions like this" actually feels reliable to developers using the plugin.