Harvard study finds OpenAI's o1-preview outperforms emergency room physicians at diagnosis

Researchers at Harvard Medical School and Beth Israel Deaconess Medical Center compared emergency room diagnoses from OpenAI's o1-preview against those from two internal medicine attending physicians. Two other attending physicians, unaware of the source, assessed both sets of diagnoses and favored the AI model.

Unlike prior AI medical studies, researchers presented each case exactly as it appeared in an electronic health record without cleaning up the data. Peter Brodeur, a study co-author, noted that AI models now score consistently close to 100% on multiple-choice medical tests, stating 'we can't track progress anymore because we're already at the ceiling.'

The researchers cautioned that AI is not ready to replace physicians. While AI excels at diagnosis, it also tends to suggest unnecessary testing that could cause harm. Additionally, there is no formal framework for accountability when it comes to AI diagnoses, and the technology falls short of gaining patient trust.