AIToday
AI in HealthcareAI Business & IndustryFortune AIPublished: Jul 29, 2026, 22:00 JST

Medical AI study finds critical flaw across OpenAI, Anthropic, OpenEvidence, Doximity

Medical AI study finds critical flaw across OpenAI, Anthropic, OpenEvidence, Doximity

3 Key Points

  1. What happened

    A new independent benchmark called NOHARM, built by researchers at Stanford, Harvard, and the ARISE network, tested AI tools from OpenEvidence, Doximity, OpenAI (GPT-5.6 Sol), and Anthropic (Claude Fable 5) on 1,100 real clinical cases with roughly 13,000 physician annotations. Doximity Ask ranked first, but the key finding was that across all systems, 76.6% of harmful errors were omissions—the AI left something out rather than stating something factually wrong.

  2. Why it matters

    Omissions in medical AI are particularly dangerous because they deprive doctors of critical information at the point of care. Eric Topol, a cardiologist at Scripps Research, noted that errors of omission must be brought close to zero, and that today's models maintain an "illusion of readiness" despite improvement. The timing is significant: the FDA loosened AI oversight in January, but states have passed more than a dozen new laws in 2026 requiring human sign-off before AI-assisted decisions reach patients, and malpractice liability remains legally unclear.

  3. What to watch

    OpenEvidence's CEO Daniel Nadler disputed NOHARM's methodology, saying the study is not peer-reviewed and questioned whether Doximity's perfect-memory approach should qualify for re-testing. The unresolved liability question—whether doctors, hospitals, or AI vendors bear responsibility when a model's suggestion is wrong—will likely shape how regulators, hospital systems, and investors approach medical AI next.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, in your inbox every morning.

Context & Analysis

The NOHARM study arrives at a critical juncture for medical AI regulation. The FDA's January decision to loosen oversight of AI-powered clinical decision-support tools came with a condition: doctors must be able to independently check the AI's reasoning. In direct contrast, more than a dozen states passed new laws in 2026 requiring human sign-off before any AI-assisted decision reaches a patient. This divergence reflects deep uncertainty about how to govern medical AI safely—neither regulators nor the legal system has settled who bears liability when an AI recommendation proves wrong. The distinction the study highlights—that omissions, not false statements, drive the most harm—cuts to the heart of this challenge. An AI that stays silent about a symptom or drug interaction cannot be fact-checked by a doctor unless the doctor already knows to look for it, undermining the assumption that human review alone can catch and correct AI errors.

OpenEvidence's rapid rise (from $1 billion(約1600億円) valuation in February to $12 billion(約1.9兆円) by January) reflects investor enthusiasm for free, evidence-based medical AI, while Doximity's enterprise model ($145.4 million(約230億円) in quarterly revenue this spring, up 5% year over year) demonstrates demand for AI tools embedded in hospital workflows with added guardrails. Yet both approaches assume that better performance scores and human oversight can resolve the omission problem. The NOHARM data suggests otherwise: even the best-performing model still leaves gaps. CEO Daniel Nadler's defense—that the study is not peer-reviewed and questions Doximity's methodology—reflects the broader tension: medical AI vendors and platforms are growing faster than the evidence frameworks and liability rules that govern medical practice can adapt.

FAQ
What is the NOHARM benchmark?
NOHARM is an independent benchmark built by researchers at Stanford, Harvard, and the ARISE network that tested AI tools on 1,100 real clinical cases and collected roughly 13,000 physician annotations to score each model for patient harm.
Which AI tool performed best in the test?
Doximity Ask came out on top in the NOHARM study. Doximity is a professional networking platform for physicians that also sells Ask, an AI assistant that helps doctors summarize patient notes, check drug interactions, and draft documentation, with a human-review layer called PeerCheck where physicians verify AI output against original sources.
What type of errors did the study find most common?
Across every AI system tested, 76.6% of harmful errors were omissions, meaning the AI left something out, not that it stated something factually wrong.

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articlePassControl gates AI agent API calls without sharing real keys