Spot Check

A spot check is a small, regular, manual read of real outputs, done as a control on the automated measurement rather than as a substitute for it. Somebody sits down, takes a handful of cases, and looks at what the system actually produced. It sounds too simple to be a practice, and it catches a category of problem that no aggregate can, because every automated number is computed by machinery that can itself be quietly broken. Scoring can silently pass everything after a change to the grader. A parsing step can drop the part of the output that mattered and score what remains. A pipeline can be evaluating a stale version. Outputs can degrade in a way the rubric does not ask about — a tone that is now brusque, a format that has drifted, a habit of answering a narrower question than the one asked. Each of these produces a healthy dashboard. Doing it well is mostly about what gets sampled. Include successes, not only failures, since the point is to verify the grader as much as the system, and a grader that is wrong is usually wrong about things it marked correct. Take the sample without choosing it. Look at the raw output rather than a rendered summary of it, because the rendering is often where information is lost. And keep it short and frequent rather than long and occasional; a handful of cases every week beats a large review every quarter, because the value is in noticing a change soon after it starts. Write down what was checked and what was seen, even when nothing was wrong. The record is what makes it possible, later, to say when a behaviour started rather than only that it exists. It is also the practice most likely to be dropped when a team is busy, and the first sign that it has been dropped is usually an incident that the dashboard had been reporting as fine for several weeks. Putting it on a named person and a fixed day survives pressure better than putting it on the team.