AI paired with humans often underperforms either alone, study finds

The research directly challenges a foundational assumption underlying most corporate AI deployment strategies: that human workers equipped with AI tools will outperform either humans or AI working alone. The eight studies identified through a structured literature search across Google Scholar, Semantic Scholar, PubMed, SSRN, NBER, arXiv, the ACM Digital Library, and IEEE Xplore represent the strongest available designs in this domain, selected using transparent criteria rather than arbitrary curation. Crucially, independent confirmation arrived through Vaccaro, Almaatouq, and Malone's 2024 meta-analysis in Nature Human Behaviour, which pooled 106 experiments (370 effect sizes) and found a task-dependent pattern: human-AI combinations underperform the better of either alone on average, with the effect reversing between decision-making tasks (where combining hurts) and content-creation tasks (where combining helps). This convergence suggests the pattern is not an artifact of the eight studies chosen but reflects what the broader empirical literature actually shows.

The medical studies illustrate the puzzle most directly. In the October 2024 JAMA Network Open trial, GPT-4 alone scored a median of 92% accuracy on clinical vignettes, while physicians using conventional resources scored lower, and physicians using GPT-4 plus conventional resources showed no statistically significant improvement over the control group using conventional resources alone. However, a follow-up trial from the same research team using a different task design found physicians using GPT-4 did achieve a +6.5 point gain (95% CI 2.7–10.2, p<0.001). The researchers acknowledge that the answer "depends on task design in ways not yet fully understood, even to the researchers running near-identical experiments."

The Harvard Business School field experiment with Boston Consulting Group introduces the concept of the jagged frontier to explain this inconsistency. Across 758 BCG consultants completing 18 realistic consulting tasks, AI-assisted performance was dramatically task-dependent: on tasks inside GPT-4's competence zone, consultants completed 12.2% more tasks 25.1% faster at measurably higher quality; on tasks outside that zone, AI-assisted consultants were 19% less likely to produce a correct solution than the unassisted control. This pattern—extraordinary on some tasks, unreliable on adjacent ones—suggests that simply adding AI access without understanding where its performance boundary lies may actively harm outcomes.