smevals is a compact evaluation framework designed to test models, prompts, and harnesses. It was developed in collaboration with Jesse Vincent’s Prime Radiant applied AI research lab to provide a simple way to compare capabilities across different model configurations.
What smevals does
- Lets users create small eval suites (directories of YAML files) that define tests and grading checks.
- Runs those evals against one or more model configurations and records runs separately from grading operations.
- Provides commands to grade recorded runs against the defined checks, browse results via a local web server, or build static HTML reports suitable for hosting.
Key commands and workflow
From the blog post, the main commands are:
- "uvx smevals docs" — outputs the README to learn about the tool.
- "uvx smevals run path-to-eval/ -m gpt-5.5 -m claude-opus-4.6" — runs the specified eval suite against multiple models.
- "uvx smevals grade path-to-eval/" — grades previous runs according to the suite's checks.
- "uvx smevals serve path-to-eval/" — starts a local web server to explore results.
- "smevals build" — generates a static HTML report from the eval results.
Example usage and intent
The author describes building an eval suite to assess how well models can write haikus as a practical example. smevals represents the author’s third iteration on an evaluation approach they've been refining for several years; they find this version promising and plan to expand it and apply it to other projects.
Why this matters
Small, configurable eval suites like smevals offer quick feedback on model behavior and the effects of prompt changes without requiring large, complex infrastructure. The YAML-based eval format and the separation of runs from grading promote reproducibility and clearer organization of results.
Tags
projects, ai, generative-ai, llms, llm, evals, jesse-vincent
