Retrieval Evaluation

Retrieval evaluation is the practice of measuring the retrieval stage of a RAG system on its own terms — did the right passages come back, and were they ranked near the top — separately from evaluating the answer the model then wrote. The separation is the entire point. When a RAG application gives a wrong answer, there are two very different causes: the necessary information was never retrieved, or it was retrieved and the model ignored or misread it. These have opposite fixes, and an end-to-end answer score cannot tell them apart, so teams that only measure the final output end up tuning prompts to repair a chunking problem. The core measurements are recall at k, precision at k, and a rank-sensitive metric such as mean reciprocal rank or normalised discounted cumulative gain. Recall at k asks whether the passages needed to answer the question appear anywhere in the top k results, and it is the metric that sets a ceiling on the whole system: information that never enters the context window cannot be used, however good the model is. Precision at k asks how much of what came back was actually relevant, which matters because irrelevant context is not free — it consumes budget, dilutes attention, and gives the model material to be confidently wrong with. Rank-sensitive metrics matter when only the first few passages realistically get read. All of this requires a labelled set: questions paired with the passages that genuinely answer them, built from real user queries rather than invented ones. Fifty to a few hundred well-chosen examples is usually enough to detect regressions, and building it is the part teams skip and later regret. Evaluation of this kind is what makes retrieval changes decidable rather than debatable, because chunk size, overlap, embedding model, hybrid weighting and reranker are all knobs whose effects are invisible without it. Practical note: pin the evaluation set in version control alongside the code and re-run it in CI on any change to chunking, embeddings or the retrieval query, and always record which embedding model produced the index — vectors from different models are not comparable, so a silent model upgrade invalidates both the index and every number you measured against it.