NVIDIA lays out a trace-driven workflow that converts production agent runs into prioritized issues, reproducible evaluations, and harness changes to improve prompts, tools, models, memory, skills and code. The NemoClaw blueprint uses three NeMo components—NeMo Compass, NeMo Eval Author and NVIDIA NeMo Relay—to demonstrate a full optimization cycle from trace analysis to measured fixes.
Why more than model tuning is needed
Agent failures often stem from harness-level issues rather than the model itself: selecting the wrong tool, following an incorrect workflow branch, missing context, using an unnecessarily expensive model, or failing to satisfy domain-specific requirements. Traces reveal where runs went wrong, and evaluations determine whether a change actually improves behavior.
Example setup and environment
The blueprint runs a Hermes Agent inside an NVIDIA OpenShell sandbox and processes recorded traces from a fictional company. In the example the Hermes enterprise assistant is connected via Model Context Protocol (MCP) to company chat, email, files and tickets. When asked to “draft a message with project updates,” the agent produced the draft but also called the send_message tool without user confirmation. Fixing this required examining both harness instructions and tool execution.
Prerequisites and recommended configuration
The tutorial expects a Linux machine with native Docker; recommended configuration is Ubuntu 24.04 with 8 vCPUs and 32 GB RAM. Required software and access include:
- OpenShell 0.1.2 and Harbor 0.22.0
- An NVIDIA API key from build.nvidia.com
- A coding agent supported by NeMo Eval Author (for example Codex or Claude Code)
A full step-by-step guide is available on GitHub, and you can also get started with an NVIDIA Brev launchable.
NeMo Compass: find what to fix first
NeMo Compass analyzes stored agent traces and evaluation evidence to surface recurring problems, identify gaps in coverage, and prioritize improvements. Instead of inspecting every run manually, Compass reduces a large trace corpus to a structured backlog of issues with supporting examples and recommended experiment areas.
In the example repository each Hermes run is recorded as one JSON trace per line; they pointed NeMo Compass at 42 traces and the ETHOS.md file that defines the assistant’s expected behavior:
compass --config configs/trace-analyst.yaml
--trace.filesystem.path traces/world-v3/production/insights.jsonl
NeMo Compass issues findings based on multiple signals: signs of unusual behavior, tool usage patterns, and divergence from intended behavior. It also returns trace IDs linking each signal to the relevant runs. In this case Compass found multiple runs where the agent sent messages without confirmation by comparing actions to the rules in ETHOS.md.
Internal benchmarks showed that running the full set of Compass signals against a dataset of thousands of traces identified anomalies that a coding agent assigned a similar task missed.
Build evaluations from real agent work with NeMo Eval Author
After Compass identifies an issue, create tests that reproduce it. NeMo Eval Author provides coding-agent skills and workflows to help generate those tests. It supports Harbor evaluations and NVIDIA NeMo Gym evaluations.
Install Eval Author’s skills into your coding agent and instruct it to use them. Provide Compass findings, raw traces, the agent’s ETHOS.md, and access to the agent repo as context. Generated evaluations should measure behavior relevant to the agent’s requirements; review and tune them to ensure they reflect those requirements.
In the example the coding agent created four Harbor evaluations with tasks like “draft a response.” NeMo Eval Author also created verifiers that check whether the agent called send_message without human confirmation and mark such runs as failures.
NeMo Eval Author can also generate custom evaluations before production data exists, using ETHOS.md requirements and code details (tool definitions and workflows). As real traces and known failures become available, expand the evaluation suite so it continues to reflect real usage.
Change one part of the harness
With an identified issue and tests in place, update the harness. In the example they changed the Hermes agent’s SOUL.md (a Hermes concept similar to a system prompt):
"Requests to write, draft, compose, prepare, or suggest a message authorize preparation only. Provide the draft for review and stop before sending it. A token returned by a preparation tool is a technical capability to execute an action. It does not establish the user's consent to that action."
Harness changes can be simple rule fixes like this or more advanced middleware, plugin extensions and other harness patterns. Give the coding agent the Compass outputs, ETHOS.md and the agent implementation so its proposals are grounded in context. Earlier posts cover harness optimizations with LangChain Deep Agents and Nous Research Hermes.
Measure before and after
After updating the harness, rerun the evaluation tasks produced by NeMo Eval Author. The example withheld two “Held out” tasks from the coding agents proposing changes to assess generalization beyond development tasks. Results were:
- Development (2 tasks × 3 runs): baseline 3/6 passes → updated 6/6 passes; tool calls 10 → 6
- Held out (2 tasks × 3 runs): baseline 4/6 passes → updated 6/6 passes; tool calls 28 → 15
The example started with a narrow set of evaluation tasks; production suites typically contain hundreds to a few thousand examples. Accept changes only when they improve both development and held-out results. If a change introduces regressions, run NeMo Compass on traces from the proposed agent and iterate. Keep held-out cases hidden from the coding agent to avoid overfitting.
Run the loop again
After a successful harness change, run the analysis again to find the next issue. In the example NeMo Compass no longer reported the send-without-confirmation finding but surfaced a new problem: for some queries the agent found an answer but failed to ground its response in retrieved evidence. That became the focus of the next round of testing and fixes.
Apply the loop to engineering agents (CAD example)
The same trace→evaluation loop applies to agents performing complex technical tasks. Evaluations should measure engineering outcomes and execution efficiency. One example uses LEO, the engineering Virtual Companion from Dassault Systèmes, to propose car-door design changes and test them via simulation to produce a stronger, safer door.
NeMo Compass identified a recurring issue where the agent placed QueryModel arguments outside the expected options field, producing an empty error message and preventing recovery. The coding agent added a validation step to catch misplaced arguments and names in tool calls. Malformed tool calls dropped from 142 to 0; accuracy remained unchanged and no regressions were observed in other scenarios.
The initial optimization run completed overnight without human intervention; in about 12 hours the reported results showed a 28% relative improvement in the door design workflow’s overall evaluation score and 35% fewer tool calls per run across six tested scenarios, with no regressions reported in other scenarios.
This example links two performance measures—task quality and execution efficiency—to the same trace-driven process: identify what to change from traces, assess the targeted improvement with custom evaluations, and check for regressions before adopting the change.
Where to start
Begin with the NemoClaw blueprint for agent optimization, the workplace assistant example or the CAD guide. Run the example agent to establish a baseline, complete one optimization cycle (find an issue with NeMo Compass, write a test with NeMo Eval Author, and try a harness change), then point the same loop at your own agent.
Summary
NeMo Compass, NeMo Eval Author and NeMo Relay form a trace-to-evaluation optimization loop: use traces to find recurring problems, build reproducible tests to measure fixes, update the harness or configuration, and rerun targeted and held-out evaluations to confirm improvements. Iterating this loop reduces recurring failures and improves both agent behavior and execution efficiency.
