Community Hackathon Reproduced Over 2,200 ICML 2026 Papers; Hundreds of Claim-Level Results Released

From July 15 to August 2, 2026, a community-run hackathon invited more than 1,200 participants to bring their own coding agents and attempt claim-by-claim reproductions of papers accepted at ICML 2026. Over 19 days the event produced 6,816 Trackio logbooks covering 2,226 papers — roughly 34% of the conference.

The exercise asked a practical question: if we re-examined a major conference at scale and tried to reproduce every paper, what would we find about reproducibility and errors? The experiment showed that coding agents can rapidly run experiments and generate reports, but humans continue to matter for steering, scale judgments, and perceptual evaluations.

How the challenge worked

The Open Reproductions challenge operated as follows:

• The organizers indexed 6,341 accepted ICML 2026 papers, extracted abstracts and the core scientific claims so agents could start from specific, checkable targets rather than 40-page PDFs.
• Participants "brought their own agent": Claude Code, Codex, Cursor, OpenResearch's orx, and other frameworks were used. The organizers provided a streamlined interface so an agent could fetch the paper, its claims, and the challenge instructions with a single command.
• Each run produced a Trackio logbook: a static Hugging Face Space containing the write-up, the code that ran, the produced artifacts, and optionally the full agent execution trace as a Hugging Face Dataset. The auditing itself was made auditable.
• An automated Logbook Judge (an open-weights model GLM-5.2) re-read every logbook and issued a per-claim verdict: verified, falsified, toy (reduced-scale evidence), or inconclusive. The judge was explicitly instructed to treat each logbook's self-assessment as untrusted.

Participants received $20 in Hugging Face compute credits for HF Jobs; across the challenge they launched 2,962 cloud jobs. Where full reproduction was impossible due to proprietary datasets or unreleased checkpoints, teams ran toy reproductions on synthetic data that mimicked the original properties.

Key numbers

• 1,221 community members joined the organization.
• 6,816 reproduction logbooks published.
• 2,226 papers attempted (34% of the conference), often by multiple independent teams.
• 35,908 claims judged; all verdicts frozen in a public dataset at the challenge close.
• 2,962 HF Jobs launched; 274 full agent-trace datasets published on Hugging Face.

Aggregate findings

• 51% of examined papers (1,103) had at least one claim independently verified.
  • 266 papers were fully reproduced (every extracted claim verified).
  • 632 papers were partially reproduced with nothing falsified.
  • In total 3,978 individual claims were confirmed with real experiments.
• 23% of examined papers (496) had at least one claim falsified or contested.
  • 49 papers had all claims falsified and nothing verifiable.
  • 242 papers produced opposite verdicts from independent teams on the same claims, illustrating that reproducibility is often adversarial rather than binary.
• Remaining outcomes: 502 papers had only toy-scale evidence, and 280 papers were inconclusive (most commonly due to missing artifacts).

Successful reproductions: examples

Some papers withstood the scrutiny well:

• "Flat Minima and Generalization: Insights from Stochastic Convex Optimization" was reproduced by 20 independent teams; 12 verified every claim. One linked logbook included the full agent trace.
• "A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness" had 14 of 17 logbooks verify every claim.

Falsifications and notable mistakes

Thirty-five participants formally claimed falsifications; the organizers adversarially re-verified every such claim by re-reading the paper and logbook, re-deriving math, or re-implementing experiments from the paper text. Confirmed examples include:

• The paging paper from the introduction: "Towards Optimal Robustness in Learning-Augmented Paging" claimed the algorithm achieves robustness H_k + O(1). One logbook measured an additive term growing like 0.38 ln k and located the exact proof step that breaks. The organizers' re-implementation extended the sweep to k = 1,024 and confirmed the growth at roughly nine sigma. The true robustness is H_k + Θ(log k).
• "Attention's forward pass and Frank-Wolfe": three independent teams found counterexamples showing token particles do not always collapse to the origin; violations first appeared at t = 224, ~3,800, and 6,416 steps. Finite-horizon checks that stopped earlier had missed these failures. The cleanest counterexample used exact rational arithmetic, eliminating floating-point ambiguity; the authors confirmed and are working on a fix.
• "Self-Distillation Enables Continual Learning": the paper's theory analyzed reverse KL divergence, but the released code's default computed forward KL. The logbook that discovered this also failed to reproduce the paper's headline +4pp result under the authors' own code and data. The authors uploaded a clarified arXiv version.
• "Do Transformers Need Three Projections?": a participant found that about 66% of evaluated label positions were EOS padding tokens that train to near-zero loss, deflating perplexity roughly threefold. The abstract claim of "3.1% quality cost for 50% cache reduction" becomes roughly 9.4% after correction.

There were also false falsifications: in one case a logbook claimed a method was 2x slower than baseline, but this was an arithmetic normalization bug (per-trajectory time compared to per-batch-of-50 time). With correct normalization, the participant's data actually confirmed the paper's claimed 8x speedup.

Contacting authors

The organizers began contacting authors for every confirmed finding with the evidence and a simple question: do you agree or is our analysis wrong? Early responses have been constructive: several authors confirmed findings, two arXiv corrections are underway, and one author had already quietly fixed an error in a new arXiv version a month before the challenge discovered it.

The role of humans moving forward

The hackathon highlighted reasons why humans remain important:

• Pure agent execution hits real limits. Agents sometimes get stuck in local loops, misread scale-dependent behavior (for example, verified verdicts on the paging paper resulted from checks that stopped before log-k growth became visible), or built falsifications on unit mismatches. The most reliable results came from workflows where humans steered the process: re-pointing agents, questioning assumptions, or deciding to abandon a flawed experimental premise instead of burning compute.
• Some evaluation remains irreducibly human. The human-in-the-loop winner illustrates this: a paper claimed stable image generation under extreme quantization. Numerical metrics suggested no collapse, but usability was a perceptual question. The agent built a review UI and a human judged all 128 image pairs; those annotations were committed and the agent validated consistency afterward. The published agent trace captures the full exchange.

The organizers frame the human reviewer role as managing intelligence effectively: setting up environments with compute, harnesses, data access, and targeted feedback so agents can do the running efficiently.

Thanks and next steps

The organizers thanked the 1,221 participants, the winners, responsive authors, and the Hugging Face and alphaXiv organizers. All logbooks, verdicts, traces, and artifacts from the challenge are public in the challenge Space. The organizers believe this is the largest open, claim-by-claim audit of a machine learning conference to date and hope it will not hold that record for long. They plan future reproduction events.