Wharton statistician Edgar Dobriban reports that a GPT-5.6 language model generated a counterexample in roughly 90 minutes to a case related to the Benjamini–Hochberg (BH) procedure that had remained open for two decades. The BH procedure is widely used to control false discoveries in large-scale hypothesis testing — from gene screens to brain imaging — and had been proven to control error rates under independence. Many practitioners had assumed the control also held under correlated data, which is common in real-world datasets.
What was found and how it was checked
According to Dobriban, GPT-5.6 produced a construction showing that the BH procedure need not provide the expected control when data are correlated. Dobriban and University of California, Berkeley statistician Will Fithian examined the result and posted the code publicly. By contrast, a month earlier the same task had been run on GPT-5.5 for 20 hours without success.
Why this matters
The Benjamini–Hochberg procedure has been cited more than 130,000 times in the literature and is a standard tool for multiple testing. If there is indeed a valid counterexample in correlated settings, that would have implications for how the procedure is applied and how results are interpreted in domains where variables are interdependent.
The model’s contribution
Reports indicate that GPT-5.6 constructed a structure not previously seen by experts: it combined asymptotic reasoning with a numerical certificate in a way statisticians described as atypical. Observers noted that while some AI-generated mathematical outputs can reflect brute-force search or extended computation, this instance appears to have produced an inventive construction within ninety minutes rather than merely exhausting a search space.
Next steps
The broader impact depends on independent verification and detailed scrutiny of the proposed example and methods. The community will likely examine the released code and attempt to reproduce the result before recommending changes to practice based on this finding.
In sum: a large language model produced, in a short span, a construction that challenges a long-standing assumption about a widely used statistical procedure; the claim was checked by Dobriban and Will Fithian and the code has been made available.
