Ai2's FairStress finds bias in how LLMs hold decisions

FairStress arrives as models are increasingly used in high-stakes settings where outputs affect people's opportunities, resources, and care—screening job applicants, approving loans, prioritizing patients for treatment, and evaluating students for admissions. Standard demographic fairness benchmarks typically test whether a model selects the same candidate in otherwise identical scenarios after demographic identities are swapped. But arriving at the same answer in two versions of a scenario can hide meaningful differences in how readily a model changes that decision when challenged.

To investigate possible sources of the asymmetry, Ai2 audited data from Olmo 3's training pipeline. Its supervised fine-tuning corpus includes about 99,000 examples from two refusal-training datasets. Across 28,000 topic clusters in the fine-tuning and preference-training data, only nine clusters discussed decisions based on merit rather than demographics—and all nine consisted of refusals. The audit also found unequal portrayals: minority-coded groups received more protective or positive framing, while white and cisgender people received little protective framing and persistent negative framing.

The benchmark's comparison with existing measures suggests the direction of measured bias depends partly on how a benchmark asks the question and scores the answer. In a controlled comparison using BBQ questions, three of four models shifted from stereotype-consistent to minority-favoring behavior when tested with a chat template, yes/no wording, and generated answers rather than fixed-option probability scoring. Ai2's findings suggest a gap between teaching models to refuse discriminatory requests and showing them how to make decisions on merit—though whether these training patterns cause the behavioral gaps remains an open question.