Offline Evaluation

Offline evaluation is the practice of scoring a model, prompt or pipeline against a fixed dataset before it reaches users. Inputs are held constant, the system under test is varied, and the results are compared. It is the counterpart to online evaluation, which measures behaviour with live traffic and real people, and the two answer different questions: offline tells you whether a change is safe to ship, online tells you whether it was worth shipping. Its value comes from being repeatable. Because the dataset does not move, two runs are comparable, a regression can be attributed to a specific change, and a failing case can be reproduced on demand. That is what makes offline evaluation the natural gate in a deployment pipeline — a prompt edit that drops accuracy on the fixed set can be blocked before it becomes an incident, and the same set run on a candidate model turns a migration from a leap of faith into a measurement. The dataset is where the quality lives. It has to be drawn from real traffic rather than imagined examples, weighted towards the cases that actually matter, and deliberately stocked with the awkward ones: ambiguous requests, adversarial inputs, questions with no good answer, and every past production failure. A test set that contains only clean inputs measures a system nobody uses. Every incident should end with a row added to this set, which is what turns evaluation into a ratchet rather than a ritual. Scoring is the other half. Deterministic checks are cheap and should carry as much of the load as possible: does the output parse, does it satisfy the schema, does it contain a required field, does it cite a source that exists. What remains is judgement, and it is scored either by human raters against a written rubric or by a model acting as judge, which is scalable but needs its own validation against human labels before its verdicts are trusted. The main risk is treating the number as the truth. An offline score is a proxy: it measures performance on the distribution captured in the set, which drifts away from live traffic as the product changes and as users learn what the system is good at. A set that is never refreshed slowly becomes an exam the system has learned to pass, and a suspiciously high score usually means examples have leaked into prompts or the set has stopped representing anything current. Run it as a suite, not as an event. Version the dataset alongside the code, run the evaluation on every change to prompts, retrieval or model configuration, keep the results next to the commit that produced them, and require a deliberate decision to ship anything that moves the score down. The point is not to reach a target number; it is to notice a change you did not intend.