AI agents can deliver large productivity gains in complex chip design workflows, but their tendency to produce plausible yet incorrect outputs is a significant barrier to production use. In multistage engineering processes, an early undetected error can silently propagate through later stages and into the final artifact. Evaluating agents against verified final outputs rather than isolated intermediate steps is therefore critical for reliable automation.
NVIDIA NeMo Gym, an open library for evaluating and improving models and agents via environments, addresses this need. The NeMo Gym agentic evaluation environment is intended to diagnose failure modes and track error propagation across complex, multistage workflows. By using a domain‑specific verifier to inspect final Liberty (.lib) files against trusted references, engineers can trace full rollout trajectories and identify root causes of subtle failures.
This article describes how the Siemens Fuse team used NeMo Gym to evaluate and increase the reliability of the Siemens Fuse EDA AI Agent when invoking characterization tools. Siemens scaled evaluations in NeMo Gym and verified the resulting Liberty files against known references and trace logs, collecting both common and rare failures across rollouts. Findings from those analyses guided changes to the agent’s skills and Model Context Protocol (MCP) tool descriptions. As a result, the measured benchmark composite score rose from about 55% to roughly 95% for a representative set of standard‑cell characterization tasks, while the number of agent turns and output tokens decreased significantly.
Automating standard‑cell characterization
Standard‑cell characterization produces timing, power, and noise models for logic cells across operating conditions. It is foundational to chip design and required when a semiconductor foundry issues a new process design kit (PDK). A PDK supplies SPICE transistor models, design rules, and PVT (process, voltage, temperature) definitions that link manufacturing to design tools. When a foundry updates a PDK, prior characterization datasets may no longer be accurate; timing and power models must be regenerated and validated for every affected cell across combinations of input slew, output load, and PVT corners.
The Solido Characterization Suite agent automates this pipeline. It orchestrates Solido tools through the Siemens Fuse EDA AI Agent, which exposes MCP servers and skill descriptions to common coding harnesses such as Claude Code, Codex, and Cursor. MCP interfaces make accelerated tools available to engineering agents, while skills guide their use in characterization workflows. Faster simulation shortens iteration cycles; NeMo Gym helps measure and improve how reliably and efficiently agents orchestrate those tools.
The Solido Characterization Suite agent must perform tool setup, large‑scale SPICE simulation, distributed compute, Liberty file generation, debugging, and final validation. Errors often yield plausible outputs rather than explicit failures, hiding incorrect thresholds, missing corners, or incomplete cells. Because these stages depend on each other, an early setup error can propagate into every subsequent step. Defining a rigorous evaluation protocol was therefore key to preparing the Solido agent for production workflows.
Measuring and improving agent reliability with NeMo Gym
As part of the NeMo collection of open libraries, NeMo Gym integrates the standard‑cell characterization flow into a controlled evaluation loop. For each rollout it defines the task, tools, state, and verifier so attempts start from a clean state and produce comparable results. It also records thinking traces, input and output messages, and tool actions.
This enabled the team to run an optimization loop:
- Run baseline rollouts with a versioned task and verifier in a custom NeMo Gym environment.
- Inspect trajectories, failed tool calls, and verifier diagnostics.
- Identify ambiguous instructions, unproductive actions, and incorrect tool selection.
- Refine the agent’s skills and MCP tool descriptions.
- Repeat the same tasks under the same evaluation conditions.
- Compare verified outcomes and efficiency per successful result.
End‑to‑end evaluation exposed failures that intermediate checks missed. Final Liberty verification revealed missing cells and PVT corners, incorrect slew thresholds, and timing or power values outside tolerance. These results highlight why engineering agents must be validated against final artifacts rather than just individual tool calls.
Reward design and artifact verification
To make results repeatable, the team implemented verifiers that output a final composite reward score composed from the following metrics:
- Coverage: whether the agent produced every expected cell and PVT corner.
- Validity: whether the delivered Liberty files are well‑formed and usable by validation tools.
- Relative validation: whether the output introduced failures beyond those present in the reference.
- Accuracy: the fraction of expected timing and power values within defined tolerances.
- Integrity: whether the workflow produced a complete result rather than reusing, truncating, or reformatting existing values.
The reward R was defined as: R = 1[Coverage] × 1[Validity] × 1[Relative validation] × 1[Integrity] × (X_tol / X_total), where X_tol is the number of output values within tolerance and X_total is the total number of output values. Each indicator 1[·] equals 1 when the gate passes and 0 when it fails; if any gate fails the composite reward is zero. Using this verifier, the team systematically evaluated and tuned the agent across multiple iterations.
Optimizing the agent and results
The team benchmarked the Solido Characterization Suite agent using the open source GF180MCU PDK. The dataset comprised 229 standard cells across 36 PVT corners and used Solido AFS for SPICE simulation. A custom agentic environment was built in NeMo Gym; for tuning runs the flow was restricted to two cells across 36 PVT corners to shorten each rollout. Each run took about 1.5–2 hours, and the team executed roughly 25 rollouts using Claude Opus 4.8 inside Claude Code as the fixed model and agent harness.
Across successive rounds the team refined characterization skills and MCP tool descriptions. To improve tool selection they simplified overlapping file and shell interfaces and defined tool inputs with explicit JSON schemas, making it easier for the agent to call characterization tools.
After these refinements, the mean composite verification score across evaluated rollouts increased from approximately 55% to about 95%. The improved workflow also required fewer agent turns and fewer output tokens: turns dropped from 155 to 53 (≈66% reduction) and output tokens fell from about 81K to 29K (≈64% reduction).
Applicability to other domains
The same evaluation approach can be applied to engineering agents in EDA, CAE, and other technical domains:
- Evaluate final engineering artifacts rather than only checking that individual tool calls succeeded.
- Implement a domain‑specific grader/verifier to detect missing, truncated, reformatted, or reused results to prevent agent shortcuts.
- Collect a large number of rollouts and iteratively optimize skills and MCP interfaces to improve reliability.
Getting started and acknowledgments
Developers can begin with the NeMo Gym Quickstart to run an initial evaluation and then apply the workflow described here: define a valid final artifact, build a domain‑specific verifier, inspect rollout failures, refine skills and tool descriptions, and rerun under consistent conditions.
Acknowledgments: the authors recognize Gunjan Shah of NVIDIA and Tim Wheler, Lakshminarayanan Ramesh, and Carl Hofmeister of Siemens for key contributions to agent development and the evaluation methodology.
