LLM résumé-screening study corrected; random noise, not bias, drove initial results

The researcher's original study, published three months ago, concluded that an auditor had flagged 45 per cent of score differences as bias in LLM résumé screening. Public feedback from three commenters—u/kamilc86, u/AssiduousLayabout, and u/hex4def6—challenged the methodology on three specific grounds: that the reasoning justifications might be invented post hoc rather than driving scores; that prompt schema ordering or other structural factors might artificially inflate variance; and that placebo prompts might inflate disagreement. Rather than dismiss these objections, the researcher ran new experiments to test each one.

The first set of experiments, involving 320 runs, showed that when positive and negative justifications were transplanted back into prompts, scores moved 3.62 points in the reasoning's direction 99.7 per cent of the time. This finding confirms that reasoning did influence scores, validating the reasoning-as-cause hypothesis. However, the same experiments revealed extreme baseline instability—meaning that LLM outputs varied wildly even for identical inputs. This instability is crucial: it suggests that much of the 45 per cent bias flagged in the original study was actually noise from this instability, mislabeled as bias.

The second and third experiments tested whether structural changes to the prompts could reduce instability and disagreement. Moving the score last in the prompt (tested across 4,800 runs) had no effect on stability and actually worsened disagreement, raising it from 33 per cent to 54 per cent. Blind prompt instructions also failed. Together, these results indicate that LLM résumé screening is unstable at a fundamental level, and that simple prompt engineering cannot fix it. The researcher's decision to revise and re-release the findings suggests that careful measurement and willingness to correct error are now shaping how this field understands LLM bias.