New LLM Position Bias Benchmark: Does an AI Judge Change Its Mind When You Swap the Answers?

Researchers tested whether large language models (LLMs) make consistent judgments or just pick whichever option appears first. They showed AI judges two slightly different versions of the same story twice—once in each order—and tracked whether the models stuck to their decision or flipped. The results are pretty damning: the median model contradicts itself 45% of the time, and GPT-5.4 is the worst offender at 66%. Even worse, most models don't just pick the first option more often—they actively rate it higher too, giving it an average bonus of +0.26 points on a 7-point scale. This reveals a fundamental flaw in how these models make decisions: they're not actually reasoning through the options fairly; they're just biased toward whatever they see first. Full benchmark data, charts, and raw outputs are available on GitHub.