OpenAI launches MentalHealthBench, 80+ experts, 22 countries

MentalHealthBench builds on OpenAI's earlier clinician-informed work for HealthBench and HealthBench Professional. The conversations were created using privacy-preserving techniques to reflect real-world usage patterns, with some scenarios including background such as a recent loss in the family so models can be tested on whether they use context appropriately. Each conversation was reviewed by at least three experts, and criteria were kept only when at least two agreed and a third did not contradict. The rubrics therefore reflect shared expert judgment rather than individual opinion.

The benchmark spans non-acute, high-acuity, and emergency scenarios, across adults, teens aged 13-17, caregivers, and clinicians. OpenAI also ran a separate analysis with 44 adults from 16 countries who had used AI for mental health or emotional support, comparing their criteria with expert guidance. That analysis did not change the benchmark's final scoring criteria, which remain based on expert consensus.

The stakes hinge on whether other researchers and developers use the open benchmark to identify gaps. OpenAI frames it as one part of broader work, alongside grants, convening experts with the Partnership on AI, and support for Transluce's mental health evaluation, as well as ChatGPT changes such as expanded crisis resources, Trusted Contact, and ChatGPT for Teens. Whether a single benchmark can shift how models handle nuanced personal conversations is likely to depend on how widely the community adopts it.