Study reveals mental health LLMs risk dangerous omissions and hallucinations, with Llama 3.3 showing 13.2% critical guidance gaps in crisis scenarios

Researchers developed UTCO framework analyzing User, Topic, Context, and Tone elements to systematically test mental health chatbot safety using 2,075 prompts

Llama 3.3 exhibited hallucinations in 6.5% of responses and omissions in 13.2%, with omissions concentrated in crisis and suicidal ideation cases

Study addresses gap in AI evaluation for mental health systems by including high-distress, narrative-based inquiries often underrepresented in existing benchmarks

Findings highlight safety risks of deploying LLMs for mental health question-answering outside clinical settings where vulnerable users seek help