The UK AI Security Institute (AISI) is publishing evaluation methods, configuration details, and verified results through the EvalEval Coalition's shared infrastructure to support more reproducible and verifiable evaluation science.
Background and earlier collaboration
AISI and EvalEval previously collaborated beginning with a joint workshop alongside NeurIPS 2025. Feedback from AISI helped shape the Every Eval Ever (EEE) schema; the current phase implements that shared infrastructure in practice.
Why reproducible evaluation reporting matters
As AI systems are deployed more widely, evaluation studies have become key evidence about model and system performance. However, results are often reported in many different formats and outlets without sufficient information to reproduce them, and rerunning evaluations can be prohibitively expensive.
EvalEval aims to improve this ecosystem through a shared reporting schema, Every Eval Ever, and an open platform, Evaluation Cards, that place evaluation results and the contextual information needed to interpret them into a common, machine-readable structure. This complements AISI's work on improving evaluation efficiency (OptStop), on increasing statistical rigor (HiBayES), and on standardisation efforts in areas such as transcript analysis and capability elicitation. Together, the organisations are diagnosing gaps in evaluation reporting and building shared infrastructure to address them.
What AISI is releasing
AISI emphasizes transcript-level transparency because it supports reproducibility, analysis, and diagnosis. Where appropriate in this next phase of the collaboration, AISI is making publicly reported evaluation methods and findings available as Evaluation Cards. The release includes verified results, context, and configuration information for the five benchmarks used in the paper's main experiment:
- HealthBench
- FrontierMath
- Humanity's Last Exam
- SWE-Bench Pro
- Terminal-Bench 2.0
These results cover six frontier models: Claude Opus 4, Claude Opus 4.5, Claude Opus 4.6, GPT-5, GPT-5.2, and GPT-5.4. The release also includes results from two related cyber evaluations — Cyber CTFs and The Last Ones — which use a different, partially overlapping set of models.
The data accompany AISI's paper "How Inference Compute Shapes Frontier LLM Evaluation," which examines how benchmark performance depends on inference-time compute and evaluation protocol.
Findings and interpretation
AISI reports that performance on Humanity's Last Exam varies with evaluation protocol and inference compute. The published curves show the cumulative share of attempted tasks solved within a given token count, using the earliest observed success per task. When models received correctness feedback from an oracle after each attempt, they continued to solve additional tasks as token use increased.
Openly released results with setup information allow researchers and practitioners to inspect individual studies more closely and to compare findings across the wider ecosystem. Where other reports omit setup details, releases like AISI's provide verified reference points for interpreting evaluations in context and for understanding how setup choices may influence reported performance. As more evaluators adopt the Every Eval Ever schema, these open comparisons can support broader and more reliable meta-research.
Invitation to contribute
EvalEval and AISI encourage participation from multiple communities:
- Model developers: report verified evaluation results.
- Evaluation developers: report benchmarks and run data using the Every Eval Ever schema.
- Evaluation, governance, and policy researchers: explore Evaluation Cards by benchmark or model, or use the platform to examine the overall state of evaluation reporting.
About the EvalEval Coalition
The EvalEval Coalition is a research community building scientifically grounded research and robust deployment infrastructure for the evaluation ecosystem. Its objective is to improve evaluation science, address the lack of consensus around documenting evaluation applicability and utility, and broaden coverage of impacts relevant for scientific research and policy analysis.
Its flagship projects include Every Eval Ever, a shared schema and repository for evaluation results, and Evaluation Cards, which combine benchmark metadata, evaluation-run data, and model metadata into interpretable records. Together these tools make it easier to understand when apparently similar scores were produced under meaningfully different conditions.
About the UK AI Security Institute
The UK AI Security Institute is a research organisation within the UK government's Department for Science, Innovation and Technology. Its mission is to equip governments with a scientific understanding of the risks posed by advanced AI. AISI conducts research and builds infrastructure to understand advanced AI capabilities and impacts, develop and test mitigations, and inform policy.
Further reading
AISI and EvalEval reference related materials including the How Inference Compute Shapes Frontier LLM Evaluation paper, HiBayES and OptStop publications, the Every Eval Ever schema, and the Evaluation Cards platform.
