Audit of 39 AI society studies finds 89.7% violate methodological principles, with reported emergent behaviors often vanishing when rigor is enforced

Researchers systematically audited 39 recent studies using large language models (LLMs—AI systems that understand and generate text) to simulate human collective behavior, identifying six pervasive flaws spanning agent profiles, interaction, memory, control, unawareness, and realism (labeled PIMMUR).

Frontier LLMs correctly identified the underlying social experiment in 50.8% of cases, while 61.0% of prompts exerted excessive control that pre-determined outcomes; when PIMMUR principles were enforced in five reproduced experiments (including a telephone game), reported collective phenomena often vanished or reversed.

The findings suggest that many observed 'emergent' behaviors are methodological artifacts rather than genuine social dynamics, raising concerns that current AI simulations may capture model-specific biases rather than universal human social behaviors.