General-purpose AI models outperformed specialized clinical AI tools across three independent tests, suggesting that widely available large language models may match or exceed the performance of proprietary medical software.

What happened

Researchers compared two specialized clinical AI tools—OpenEvidence and UpToDate Expert AI—against three frontier general-purpose large language models (GPT-5.2, Gemini 3.1 Pro, and Claude Opus 4.6) using three evaluation stages: 500 medical knowledge questions, 500 clinician-alignment items, and 100 real clinical queries reviewed by 12 US clinicians. On medical knowledge questions, Gemini achieved 97.4% accuracy, GPT 94.2%, and Claude 90.2%, while the clinical tools scored 89.6% (OpenEvidence) and 88.4% (UpToDate). Frontier models also outperformed clinical tools on clinician-alignment scoring and real-world clinical queries.

Why it matters

Specialized clinical AI tools are entering medical practice at scale, yet their internal design and training remain proprietary—making it difficult for clinicians and health systems to assess their value independently. This study provides the first rigorous, blinded comparison showing that general-purpose models available to anyone may perform as well as or better than tools marketed specifically for clinical use, challenging the assumption that domain-specific AI tools offer superior clinical performance.

What to watch

The clinical tools performed comparably to auto-enabled Google Search AI Overview on real clinical queries, suggesting that physicians may be relying on similarly capable (and more widely accessible) tools already. The research highlights the need for independent evaluation before specialized clinical AI tools enter medical settings at scale.