SOTOPIA-TOM benchmark evaluates how LLM agents handle private information and coordination in multi-party settings

Researchers introduced SOTOPIA-TOM, a benchmarking framework with 160 human-reviewed scenarios across eight industry sectors, each involving 3 to 5 agents with partitioned private knowledge and channel-dependent sharing policies. The framework measures how well agents share useful information, seek missing details, coordinate efficiently, and protect privacy.

Results across 6 LLM backbones show that even the largest high-reasoning model (GPT-5) reaches only a 62% INFOMGMT score, indicating persistent deficiencies in information seeking and privacy-aware decision-making. ToM-based interventions reduced critical privacy violations on GPT-4o from 9.9% to 2.2% while increasing the composite InfoMgmt score more than 2.5x from 15% to 40%.

The framework enables both public (broadcast) and private (direct message) communication, and uses a multi-dimensional evaluation system to assess interaction abilities in information-asymmetric and privacy-sensitive settings.