Irene Solaiman, chief policy officer at Hugging Face, told TechCrunch Disrupt that methods behind model benchmarking are complex and still evolving. She argued that testing needs to become more consistent if the industry is to rely on these evaluations going forward.
Trust, reproducibility and model access
Solaiman noted that “trust” is a loaded term and highlighted reproducibility as a central concern: can evaluation results be reproduced? She emphasized that adequate access to the model — notably access to open weights — is often required to reproduce outcomes. Reproducibility has long been one of the stickiest issues in benchmarking and remains a subject of active debate.
Comparability and a thin ecosystem
Beyond the validity of an individual test lies the question of how different evaluations compare to one another. Solaiman described evaluations as an evolving science for which there is not yet a reliable way to meaningfully compare results across tests. She warned that many evaluations lose effectiveness relatively quickly and that the ecosystem supporting evaluators is not as robust as it needs to be.
Embedded evaluators: popular idea, uncertain effectiveness
The idea of embedded evaluators — nominally neutral third parties placed inside AI organizations to review systems — has gained traction in recent months as labs respond to a series of safety incidents. Hugging Face itself was a notable target of one such incident. Rather than rely solely on government regulation, many labs have pushed for third‑party evaluation, but doubts persist about how neutral or capable these evaluators will be in practice.
Solaiman pointed to Anthropic’s decision to hire Accenture as its first evaluator as an example that provoked backlash and skepticism over whether a consulting firm was qualified for that role.
From model tests to system‑level and agentic evaluations
Evaluations are shifting away from testing models in isolation toward assessing the whole system around a model, including the harness (the tool or interface used to access the model) and agentic setups. Solaiman said the harness heavily influences performance: a weaker model with a superior harness can outperform a frontier model.
She added that agentic evaluations are particularly difficult and costly. While there are increasingly many evaluations for models, far fewer exist for harnesses, and the field is only beginning to develop better tests for holistic agents — the parts of the system that deliver most practical value.
Conclusion: need for standardization
In short, the current evaluation landscape is somewhat of a free‑for‑all. If benchmarking is to remain a useful tool for industry leaders and regulators, Solaiman argued that a more uniform, industry‑wide approach and stronger evaluator ecosystem will be necessary.
