Study finds identity-dependent scoring bias in multi-agent LLM evaluation pipeline TRUST; full anonymization required to detect it

Researchers measured scoring bias across the TRUST democratic discourse analysis pipeline by testing four model families with two anonymization scopes across 30 political statements. Single-channel anonymization produced near-zero bias effects because individual channels act in opposite directions and cancel each other out, masking the true pattern of identity bias.

Full-pipeline anonymization revealed that homogeneous model ensembles amplify identity-driven sycophancy (preference favoring the same model type) when model identity is visible, while heterogeneous ensembles show the reverse effect. One tested model exhibited baseline sycophancy two to three times higher than others and near-zero deliberative conflict on ideological topics.

The findings indicate that partial anonymization is insufficient and actively misleading for bias measurement. Heterogeneous model ensembles are structurally more robust than homogeneous ones, achieving higher consensus rates and lower identity amplification. Systems validated under partial anonymization or with homogeneous ensembles may pass validation while retaining invisible structural identity bias.