NVIDIA has released SkillEvaluator, an open-source evaluation layer designed to quantify how packaged “skills” change AI agent behavior and outcomes. Skills bundle instructions, examples and tool guidance so agents can move more directly from user intent to a solution with fewer wasted steps and less token use.
What is a Verified Skill?
Verified Skills are NVIDIA-packaged and signed capability descriptors that specify what an NVIDIA product does, when to invoke it, and how to call it. The “verified” label is assigned after measurements show the skill is ready for use.
Evaluation methodology
SkillEvaluator evaluates skills through three tiers, each of which can run independently:
- Tier 1 — Safety and structure: static checks such as schema and frontmatter validation, quality scoring, prompt-injection and data-exfiltration scanning, secret and PII detection, license checks, and script linting.
- Tier 2 — Distinctiveness: embedding-similarity analysis to find duplicated guidance inside a skill and overlapping coverage across the catalog.
- Tier 3 — Live evaluation: run live, sandboxed comparisons in which an agent executes generated tasks once with the skill installed and once without it, and the difference is measured.
For Tier 3, SkillEvaluator uses Harbor, an open-source framework that runs agent evaluations in repeatable, isolated environments. SkillEvaluator prepares the Harbor setup, converts evaluation cases into tasks, executes agents in sandboxes, gathers results, and computes the skill’s impact.
Each Tier 3 result is a controlled comparison: the agent harness runs each evaluation case twice (with and without the skill) using the same prompt, model, task inputs, and grading rules. The only experimental variable inside a harness is the presence of the skill. Each skill is evaluated across two independent agent harnesses; the difference in scores (with-skill minus without-skill) is reported as Skill Lift in points.
Example workflow
A typical workflow: generate an evaluation dataset (skillevaluator create-eval-dataset ./my-skill --full) which produces evals/evals.json containing IDs, prompts, expected outputs and optional assertions. Then run the Tier 3 comparison (skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker). SkillEvaluator packages the cases for Harbor, runs each case with and without the skill, grades both runs, and reports scores and Skill Lift.
Live evaluation results — measurement basis
The published figures in this report use the benchmarks.json snapshot from the NVIDIA/skills repository dated August 12, 2026 (commit 738d79e). Scores are macro-averaged so every skill–harness pair counts equally. Skill Lift is the numeric difference between with-skill and without-skill scores. The catalog is evaluated continuously and current numbers are available in the repository’s benchmarks.json.
Baseline (without-skill) results
In the snapshot, without-skill averages from Codex and Claude Code runs are (score out of 100):
- Correctness: 46
- Discoverability: 42
- Effectiveness: 39
- Efficiency: 43
- Security: 97
Security stands out with a high baseline because the primary objective in that dimension was to confirm that installing a skill does not introduce a regression.
Known limitations in interpreting baselines
Correctness, Effectiveness and Security measure run outcomes, so their baselines indicate what the agent achieves without the relevant skill. Discoverability and Efficiency also evaluate how a skill is discovered and used; without a skill those actions are unavailable, but agents can still get credit for productive tool use or for correctly leaving a non-relevant skill unloaded. Most skills were evaluated with a single attempt (85% of published results were one attempt, 15% were two), so individual skill scores vary; catalog-wide averages aggregate thousands of trials but confidence intervals are not reported here.
With-skill results and Skill Lift
Using skills from the NVIDIA/skills GitHub, average with-skill scores and Skill Lift relative to the without-skill baseline were:
- Correctness: without 46 → with 87 | Skill Lift +41
- Discoverability: without 42 → with 82 | Skill Lift +40
- Effectiveness: without 39 → with 78 | Skill Lift +39
- Efficiency: without 43 → with 78 | Skill Lift +35
- Security: without 97 → with 98 | Skill Lift +1
Across all dimensions the mean Skill Lift is +31 points; excluding Security the mean Skill Lift is +39 points. These increases indicate verified skills substantially improve average performance on the specialized tasks evaluated, especially for correctness, discoverability and end-to-end effectiveness. Discoverability and Efficiency gains should be read as evidence the agent activates and uses the verified skill correctly when it is present — not as measures of unaided behavior.
Skill Lift by harness
Skill Lift varies by harness:
- Claude Code — Skill Lift across all dimensions: +34; excluding Security: +42
- OpenAI Codex — Skill Lift across all dimensions: +29; excluding Security: +36
Differences are expected because harnesses use different default system prompts, context handling, and tool-calling implementations. The verified skill provides structured grounding neither harness supplies by default.
Partner pilots and integrations
Several partners integrated SkillEvaluator in pilot deployments:
- OpenClaw is piloting SkillEvaluator for organizations on ClawHub, running Tier 3 evaluations and showing with-skill and without-skill results in an Evals tab for developers.
- Nous Research tested SkillEvaluator inside Hermes Agent with an optional SkillSpector advisory scan during skill installation. SkillSpector checks for PII, Unicode smuggling, script linting, license and security issues and surfaces file-line findings before installation. This workflow is covered by 29 passing tests; each skill scan takes about 1.4–1.5 seconds.
Key findings
NVIDIA highlights three practical takeaways for teams building and testing skills:
- Better evaluation datasets yield better skills: clearly specifying tasks, expected outputs and out-of-scope requests sharpens evaluation signals before any agent runs.
- The product (domain and task) matters more than the agent: Skill Lift varies far more across products than across harnesses; per-product Skill Lift ranged roughly from +2 to +46 points.
- Token savings are not automatic: SkillEvaluator separately tracks token usage. Two single-attempt examples from the NVIDIA verified skills repository illustrate this: the jetson-optimize-memory skill reduced tokens from 617,306 to 142,540 (a 76.9% reduction) and execution time from 474.9 to 220.0 seconds (53.7% reduction). In contrast, cuopt-install increased tokens from 25,227 to 55,582 (120.3% increase) and execution time from 34.0 to 41.1 seconds (20.8% increase), indicating optimization opportunities.
Getting started
Developers can consult the SkillEvaluator documentation and review the verified skills catalog on NVIDIA’s GitHub to examine skill definitions and benchmark data.
Acknowledgments
NVIDIA thanks Roshni Malani, Meghana Puvvadi, Subodh Prabhu, Mohit Gupta, Yogesh Dangi, Yashraj Basaravaj Patil, Keshav Pradeep, Siddharth Itagi, Alejandro Sanabria Portala and Pranita Maske for their contributions.
