Position Bias

Position bias is the tendency of an LLM to be swayed by where information sits rather than what it says — most visibly when the model acts as a judge or picks among options. In pairwise evaluation, judges often favor whichever candidate is presented first (or sometimes last) regardless of quality; the same effect shows up in multiple-choice tasks, where reordering the options changes the answer. It was documented prominently in LLM-as-judge work such as MT-Bench (Zheng et al., 2023) and is related to, but distinct from, 'lost in the middle,' where facts buried mid-context get underused. It matters because it silently corrupts any ranking, routing, or selection you build on top of a model, producing results that look principled but track order. For builders, the fixes are cheap: swap positions and average both orderings, randomize option order, and calibrate against a small human-labeled set. If a decision flips when you only reordered the inputs, position bias is doing the deciding — not the content.