GeneBench‑Pro is a research‑level benchmark intended to evaluate whether AI agents can make the higher‑order judgments required by real computational biology work. It extends the original GeneBench to include more difficult, realistic problems across genomics, quantitative biology, and translational medicine, aiming to reflect the complexity, iteration, and ambiguity of scientific research.
Why this benchmark matters
Scientific data rarely arrive with detailed instructions. Researchers must decide if a pattern reflects biology or noise, whether the data can answer the posed question, and whether results are decision‑ready. Existing assessments have rarely measured the system‑level judgment calls that make real computational research hard—handling ambiguity, revising assumptions, choosing analysis paths, and recognizing when a result is fit for decisions.
GeneBench‑Pro is built to measure these higher‑level capabilities precisely. The project frames “research taste” as chains of judgment that determine an analysis: which questions the data can support, how early diagnostics should change the model or estimand, and when an initial plan should be revised.
Benchmark design and structure
- The suite contains 129 problems spanning a wide range of computational biology settings and methods.
- Each problem provides a realistic, messy dataset, a brief experimental context, and a target estimand tied to a downstream decision. Correct solutions require data exploration, selection of an appropriate analytic approach, iterative experimentation, and a final numeric answer.
- Problems are synthetic: the full causal structure is known and the data‑generating process is simulated. This lets the creators tune problem complexity, ensure reasonable subjective analytic choices still yield acceptable numeric results, and verify (via ablation studies) that plausible but incorrect analyses fail.
- Problem drafts are audited with detailed trace analyses to check for information leakage and unintended solution paths, increasing confidence that correct answers require the right analytic pathway rather than shortcuts or author preferences.
External review and openness
82 of the 129 GeneBench‑Pro questions were sent to external domain experts—graduate students, postdoctoral researchers, industry scientists, and professors—for review. Reviewers judged realism, identifiability of the target answer, and appropriateness of methods and estimators; the feedback was used to improve problems.
Ten representative problems are being open‑sourced on Hugging Face with an interactive interface. A 50‑question subset will be provided to Artificial Analysis for independent third‑party benchmarking in the near future.
Execution environment and example problem
Each problem is a self‑contained scientific analysis. Agents are given an isolated workspace with a short prompt, data files, and a standard bioinformatics stack including Python, scientific computing libraries, and basic genomics packages like PLINK 2.0, although the tasks do not require domain‑specific tooling.
One illustrative problem draws from a molecular tumor board registry of trial‑eligible advanced solid‑tumor cases considered for a TXR1‑directed inhibitor. The task: for tumors with SV‑driven TXR1 target‑mediated activation at time zero, estimate the marginal effect of TXR1i versus non‑TXR1 systemic therapy on week‑16 clinical benefit as if all patients had an assessable week‑16 visit, and estimate the 8‑week treatment‑limiting toxicity/discontinuation risk under TXR1i in the same target population. Compute net clinical utility as benefit risk difference (percentage points) minus 0.35 times toxicity risk (percentage points), and choose therapy_class_code 1 if TXR1i has positive net utility and 0 otherwise. The problem notes that the data come from a real experiment and grades not only numeric correctness but analytical reasoning quality; it also requires returning a final answer as exactly one JSON object.
Avoiding common benchmark failures
Many multi‑step biological benchmarks build questions around historical messy datasets, creating ambiguity about correct analytic paths or numerical insensitivity that lets incorrect analyses pass. GeneBench‑Pro mitigates these failure modes by using synthetic, auditable data with verified causal structure so that passing depends on correct analytic choices rather than arbitrary cutoffs or shortcuts.
Evaluation results
Because the data generation is controlled, correctness can be graded deterministically against known targets, eliminating variability from rubric‑based evaluation.
Our strongest model, GPT‑5.6 Sol, achieves a pass rate of 28.7% at the highest reasoning level (31.5% with Pro mode enabled). This is a sharp improvement over the original GeneBench development stage, when GPT‑5 scored below 5%. However, at the lowest reasoning level GPT‑5.6 Sol attains only a single‑digit pass rate. Increased test‑time compute improves performance substantially: at the highest reasoning level GPT‑5.6 Sol solves nearly six times as many questions as GPT‑5.2 while using about two‑thirds as many tokens.
Comparisons across model families indicate GPT models are among the strongest at high‑level scientific reasoning under quantitative uncertainty. The performance gap between GPT‑5.6, GPT‑5.5 and leading open‑source models like GLM 5.2 is larger than expected from coding benchmark extrapolations, suggesting open‑source models remain more specialized for coding than for broader reasoning.
Frontier GPT models were used during development to evaluate and harden problems; we therefore considered the possibility of bias toward GPT models. In practice, competitor models at best matched the corresponding GPT model at release and generally fell short.
Economic and scientific implications
Reviewers estimated a typical GeneBench‑Pro problem would take a human expert about 20–40 hours to complete. At a conservative $200/hour, the human labor cost of a single problem is in the thousands of dollars. Current AI agents are still too unreliable to fully replace human experts, but inference costs are only a few dollars per problem; thus partial automation at current capability could yield meaningful economic and scientific value.
Remaining gaps and outlook
Frontier models still solve fewer than a third of the problems, leaving substantial room for improvement. Models can make partial progress on challenging tasks but struggle to close the inferential loop—mirroring differences between novices and experienced experts, where experience helps frame problems and adapt approaches, while novices have trouble integrating observations into broader context.
Some example methodological differences highlighted by the benchmark include handling treatment timing with conventional Cox outcome models without addressing treatment‑confounder feedback versus using new‑user marginal structural Cox models that exclude 818 flagged prevalent users, model treatment initiation with stabilized inverse‑probability weights, and treat exposure as time‑varying with a 90‑day efficacy lag.
Benchmarks like GeneBench‑Pro aim to convert vague capability gaps into diagnosable, fixable problems by reliably measuring progress and pinpointing failure modes. If agents can reliably automate this class of analysis, they could accelerate scientific discovery: human genetic evidence already shapes target prioritization, sequencing costs have fallen, and biobank‑scale datasets now link molecular, phenotypic and health‑record information at unprecedented scale. The bottleneck is shifting from data generation to turning information into actionable insights, and models that can perform analyses now handled by expert teams could transform industrial research by speeding hypothesis triage, target follow‑up, and the data‑decision iteration cycle.
GeneBench‑Pro is an initial effort to evaluate the abstract skills of good scientific judgment; as model capabilities advance, benchmarks probing these higher‑level abstractions should become more valuable than those that only test book knowledge or routine analysis execution.
