GPT-5.6 Sol disproves 30-year statistics conjecture in 90 minutes

Edgar Dobriban, an associate professor at the University of Pennsylvania's Wharton School, set out to tackle one of the central open questions in statistics using OpenAI's GPT-5.6 Sol Pro. The problem centered on the Benjamini-Hochberg (BH) procedure, a method developed in 1995 by Yoav Benjamini and Yosef Hochberg to address a recurring challenge in statistical testing: when researchers test thousands of hypotheses simultaneously—such as scanning the human genome for disease-linked genes—false positives proliferate. The BH procedure controls the false discovery rate, or FDR, defined as the share of reported significant results that are actually false alarms. The original paper has accumulated more than 130,000 citations and is now widely deployed across modern statistics and scientific fields. Benjamini and Hochberg originally proved their method works with independent data. Real-world data, however, is often correlated: genetic variants, for instance, are frequently linked when certain genome locations are inherited together. For decades, statisticians assumed the BH procedure would also work reliably with correlated, normally distributed data, particularly when testing for deviations in both directions. Yet despite its widespread use and theoretical importance, no one had ever formally proved this assumption.

Dobriban used GPT-5.6 Sol Pro to construct a statistical model where the actual false discovery rate provably exceeds the target level, disproving the long-held conjecture. Simulations confirmed the result, and Dobriban published the accompanying code alongside a preprint. The gap is modest: he reports the actual false discovery rate as 0.104 versus the target of 0.1. Dobriban wrote that this result "mainly matters for theory at this point" and that practical effects require further investigation. Critically, the finding does not imply the BH procedure is unusable in practice.

What captured attention was the speed and capability gap. GPT-5.6 Sol Pro solved the problem in about 90 minutes. Its predecessor, GPT-5.5, could not find a solution even after roughly 20 hours of work with several agents. Dobriban remarked on this stark difference: "So the capability improvement is quite real. Exciting times to live in!" Berkeley statistician Will Fithian, who works in the same area, called the disproved conjecture "the most interesting open problem in my area of statistics" and characterized the result as "another marker of advancing AI capabilities whose consequences will reach far beyond math." Fithian also reflected on the emotional weight of the moment, writing: "I can't help but mourn the bygone days when a key result always meant a colleague to celebrate; a human insight to admire; a human achievement to be inspired by."

The solution's mechanism reveals both promise and limits. Dobriban observed that GPT-5.6 Sol Pro combined known statistical methods in an unusual configuration rather than inventing entirely new ones. He noted the combination was unusual but ultimately "not especially surprising." The challenge lay in finding the right way to connect existing approaches—a task the newer model proved adept at solving. This observation touches on a deeper question unresolved by the result: can AI systems trained on human data reason their way to genuinely novel knowledge, or are they constrained to recombining patterns from their training? Even if recombination is their limit, Dobriban's work demonstrates practical utility as a tool integrated into human research workflows. However, more ambitious objectives—such as building self-improving AI that can generalize—may require capabilities beyond recombination, a possibility that has drawn the interest of deep learning pioneer Richard Sutton, who recently founded a startup to pursue exactly that goal.

The Benjamini-Hochberg procedure has been a cornerstone of modern statistics since 1995, particularly in fields like genomics where researchers must filter false positives from thousands of simultaneous tests. For nearly three decades, the statistical community assumed the method would work reliably even when data points are correlated—a common real-world scenario, such as when genetic variants are inherited together. However, no one had formally proved this assumption. The University of Pennsylvania's Edgar Dobriban leveraged GPT-5.6 Sol Pro to fill that gap, constructing a counterexample showing that under certain conditions, the false discovery rate can exceed the target level. The speed of the solution is what stands out: 90 minutes versus roughly 20 hours for the previous generation model, and decades of unsuccessful human effort. Dobriban acknowledged that the AI's approach combined existing statistical methods in an unusual way rather than inventing new ones—a pattern seen in similar AI breakthroughs in mathematics. This raises a fundamental question about AI's reasoning: can these systems generate truly novel knowledge, or are they limited to sophisticated recombination of ideas learned during training? Even if recombination is the ceiling, Dobriban's work demonstrates practical value as a tool embedded in human workflows. The broader implications remain uncertain, though some researchers, including deep learning pioneer Richard Sutton, believe genuine self-improvement and generalization require capabilities beyond recombination.

No comments yet. Be the first to share your thoughts!