Every Eval Ever (EEE) and Hugging Face Community Evals can now interoperate: EEE’s standardized JSON evaluation records can be converted into the small YAML files the Hugging Face Hub expects. The integration aims to make evaluation results visible where people look at models while preserving a link back to the full, structured EEE record.
Why this matters
Evaluation results are how model capabilities are measured, models are compared, and safety and governance decisions are reasoned about. Today, results are scattered across papers, leaderboards, blog posts, and harness logs in many different formats. The same model on the same benchmark can report different scores depending on who ran it and how—for example, LLaMA 65B has been reported at both 63.7 and 48.8 on MMLU. Many discrepancies stem from evaluation settings that are commonly unreported.
EEE launched in February 2026 as a project of the EvalEval Coalition, while Hugging Face launched Community Evals in February 2026 to decentralize how benchmark scores are reported on the Hub. Combined, the projects reduce uncertainty about how users, researchers, and policymakers trust, interpret, and choose evaluations and models.
What the EEE schema records
EEE provides one JSON schema for reporting an evaluation result. It records, among other things:
- who ran the evaluation
- which model was used
- how the model was accessed
- generation settings
- what the metric actually measures
- an optional companion JSONL for per-sample outputs
The schema was developed with feedback from researchers and policy experts and accepts results from any source (harness logs, leaderboard scrapes, paper numbers, etc.), so these different sources end up in a common shape. The GitHub repository contains converters, examples, and a contributor guide.
Scale and cost considerations
The EEE datastore on Hugging Face has grown to around 229,000 evaluation results across more than 22,000 models and 2,200 benchmarks, pulled from 31 different reporting formats. Reproducing those runs from scratch would cost on the order of hundreds of thousands of dollars, which supports the argument for preserving and standardizing results once they have been generated.
How the two systems work together
Hugging Face Community Evals has two sides:
- A benchmark lives in a dataset repository and registers itself with an eval.yaml. The dataset page then collects and displays a leaderboard of every score reported against it across the Hub. The list of official benchmarks grows over time.
- A model’s scores live in .eval_results/*.yaml inside the model repository. These appear on the model card and feed into the matching benchmark leaderboard. Both the model author’s own results and results submitted by others via PRs are aggregated, and each score carries a badge indicating whether it was author-submitted, community-submitted, or independently verified.
When you send a result to both systems, two things happen: the score appears on the Hugging Face model page and in the benchmark leaderboard, and it carries a source badge that links straight back to the full EEE JSON record, where the generation config, harness version, reproducibility notes, and any instance-level data live.
If you submit through your organization’s official Hugging Face account, your results show a verified checkmark on EvalEval, signaling that the numbers come directly from the source.
The converter and how it works
The central tooling is the community eval converter, which translates EEE records into the YAML structure Community Evals uses. The converter maps fields such as:
- source_data.hf_repo -> dataset.id
- evaluation_name -> task_id
- score_details.score -> value
- evaluation_timestamp -> date
It also inserts the EEE datastore object URL into the source block so the Hub entry backlinks to the per-record EEE JSON. The converter currently handles four official benchmarks: MMLU-Pro, GPQA, HLE, and GSM8K.
Beyond field mapping, the converter:
- downloads the specified EEE datastore collection and referenced records, verifies object hashes, and finds scores that map to supported benchmarks;
- audits what already exists in the model’s .eval_results YAML files on the main branch and in open PRs, comparing by dataset and task rather than by filename;
- labels entries as already_present, score_conflict, missing_hf_model, or ready;
- writes local YAML previews and a review file, shows a readiness report, and only opens PRs after the user types OPEN PRS and supplies a commit message.
Nothing is pushed without the user’s explicit sign-off. Reruns reuse cached collection results unless the --force flag is passed.
How to get started
- Submit your full records to the EEE datastore.
- Use the community eval converter tool from the GitHub repository. Example command to process a collection:
uv run tools/hf-community-evals/community_evals_converter.py MMLU-Pro
--datastore evaleval/EEE_datastore@main
- Review the generated previews and report, then type OPEN PRS when you’re ready to submit. Full documentation for the schema, CLI, and converters is available at evalevalai.com/every_eval_ever/hf-community-evals.
Summary
Linking EEE and Hugging Face Community Evals makes the same evaluation simultaneously visible on the Hub and traceable to a complete structured record, reducing fragmentation, improving transparency, and supporting reproducibility without forcing manual format duplication.
