Inter-Annotator Agreement

Inter-annotator agreement measures how often two or more people independently give the same label to the same example. It is the health check on a labelling task, and it should be run before any large annotation effort rather than after. If two trained annotators disagree on a quarter of the examples, the task itself is under-specified, and every downstream number — training accuracy, evaluation scores, model comparisons — inherits that ambiguity as noise that no model can resolve. Raw percentage agreement overstates the picture, because two annotators labelling a heavily imbalanced set will agree often by chance alone. Chance-corrected statistics such as Cohen's kappa for two raters or Krippendorff's alpha for more are the standard alternative; what they report is agreement beyond what random labelling of the same distribution would have produced. There is no single universal threshold, and quoting one is usually a mistake — the level you need depends on the cost of an error in your application. The right response to low agreement is almost never to average the labels. It is to read the disagreements, find the categories that overlap or the instruction that does not cover a real case, rewrite the guideline, and re-measure. In LLM evaluation the same discipline applies to model judges: before trusting a model to grade outputs, check its agreement with human raters on the same sample, and treat the gap as the error bar on every score it produces afterwards.