ChatGPT health advice worse for free users, better for paying subscribers

OpenAI's Health feature launch represents a deliberate two-tier approach to AI-assisted medical guidance, with the company explicitly restricting its most capable health model to paying subscribers. The benchmarks OpenAI uses show substantial performance gaps—GPT-5.6 Sol beats both physician-written answers and the free-tier model across every category tested. However, the company's own caveats reveal the limits of these results: benchmark tests measure knowledge in artificial environments and cannot replicate the diagnostic work of in-person medical exams, where doctors assess nonverbal cues and draw on clinical experience. OpenAI's disclosure that more than 260 physicians helped develop the feature suggests effort toward medical credibility, yet the company simultaneously warns that ChatGPT still makes mistakes and cannot replace medical advice.

The practical reality is more nuanced than the performance numbers suggest. Recent research on AI radiology systems (RadLE 2.0) found that none of 16 AI models tested performed as well as human radiologists, often because chatbots express incorrect findings with high confidence rather than acknowledging uncertainty. At the same time, other systems like MIRA and AMIE have demonstrated performance comparable to primary care doctors in simulated consultations, suggesting AI can support routine tasks under physician oversight. OpenAI's early testing revealed that over 70 percent of users asked health questions outside the dedicated Health section because switching between modes was cumbersome—a friction the company has now reduced by making Health available in any conversation. The feature's absence from Europe, tied to stricter data privacy and potential AI Act classification, underscores regulatory tensions the company faces globally. With over 300 million people now asking ChatGPT health questions each week (up from 230 million in January), the scale of this rollout and its tiered quality model will likely shape how millions approach preliminary health research.