Researcher releases benchmark to measure position bias in AI language models — shows which chatbots favor answers based on order rather than quality

A GitHub repository published a test framework called LLM Position Bias Benchmark that measures whether AI language models (systems like ChatGPT that generate text) rank answers differently depending on which option appears first, second, or third — a flaw that makes their judgments unreliable when evaluating two pieces of text side-by-side.

The benchmark uses "swapped-order pairwise judging" — it asks the same AI model to compare two texts multiple times, but switches which one appears first each time. If the AI's ranking flips just because the order changed, it reveals the model is biased toward position rather than actual quality, similar to how humans sometimes favor the first option they see.

For product managers and developers building AI tools that need fair comparisons (content moderation, hiring screeners, academic paper review systems), this benchmark exposes a critical blind spot: your AI may be ranking things based on presentation order, not merit. Teams can now test their own models before deploying them to catch this bias.