NVIDIA NeMo Gym and Siemens: Chip-Design Agent Score Rises From 55% to 95%

NVIDIA and Siemens EDA have published a concrete recipe for making an engineering agent reliable: grade its final output files, read the failed rollouts, then rewrite its skills and tool descriptions. In the NVIDIA Technical Blog post of October 14, 2026, the authors say the loop lifted the mean composite verification score of Siemens' Solido Characterization Suite agent from about 55% to about 95%, while turns fell from 155 to 53 and output tokens from about 81K to 29K.

The numbers are the headline, but the method is the useful part. It applies to any agent that produces a checkable artifact, from a netlist to a config file to a data migration.

Quick reference

The problem: plausible but wrong chip data

Standard-cell characterization produces timing, power and noise models for logic cells across operating conditions. It is needed whenever a foundry issues a new process design kit (PDK), because existing characterization data may no longer describe the updated devices. Models must be regenerated and validated for every affected cell across combinations of input slew, output load and process, voltage and temperature (PVT) corners.

The Solido agent handles tool setup, large-scale SPICE simulation, distributed compute, Liberty file generation, debugging and final validation. NVIDIA's authors stress why this is risky: errors often produce plausible output rather than a clean failure, hiding "incorrect thresholds, missing corners, or incomplete cells", and an early setup error can propagate into every later step. Siemens exposes MCP servers and skills to coding harnesses including Claude Code, Codex and Cursor.

How NeMo Gym is used

NVIDIA calls NeMo Gym an open library, part of its NeMo collection, for evaluating and improving models and agents using environments. Its documentation describes an environment as a task dataset, an agent harness, a verifier that scores completion, and per-task state, and the project's catalog page warns it is in early development, so expect evolving APIs. Here it defines the task, tools, state and verifier for each rollout so every attempt starts clean and results are comparable, and it records thinking traces, messages and tool actions.

The optimization loop, as the post lists it:

1. Run baseline rollouts with a versioned task and verifier in a custom NeMo Gym environment.
2. Inspect trajectories, failed tool calls and verifier diagnostics.
3. Identify ambiguous instructions, unproductive actions and incorrect tool selection.
4. Refine the agent's skills and MCP tool descriptions.
5. Repeat the same tasks under the same conditions.
6. Compare verified outcomes and efficiency per successful result.

The diagram above shows it: standard cells go into the agent, the agent writes Liberty files, an artifact verifier produces verified timing files, and trajectory analysis feeds updated skills back to the agent. NVIDIA says the whole process is driven by a coding agent with a human in the loop.

The reward: four gates and a tolerance fraction

The key design decision is to verify the final artifact. The team's verifier gives a composite reward built from:

The reward is the product of the four gates (each 1 or 0) times the fraction of values within tolerance, so any failed gate zeroes the score. The integrity gate is the anti-cheating part: it targets an agent that copies reference values or truncates work to look finished. The authors note end-to-end checks surfaced failures that intermediate checks missed, such as missing cells and operating conditions, wrong slew thresholds, and values outside tolerance.

This connects to a theme we have covered before: agents that game checks. See our FrontierCode 1.1 maintainer-merge benchmark coverage, which also grades outcomes rather than steps.

The setup and the results

The team used the open-source GF180MCU PDK, whose dataset has 229 standard cells across 36 PVT corners, with Solido AFS for SPICE simulation. For tuning, each run was restricted to two cells across 36 corners, and each run took roughly 1.5 to 2 hours. About 25 rollouts used Claude Opus 4.8 in Claude Code as a fixed model and harness, so model choice did not confound the result.

Changes made across rounds included simplifying overlapping file and shell interfaces and defining tool inputs with explicit JSON schemas.

The post's table reports this for Claude Opus 4.8 only, as development results before and after harness optimization.

Why this matters for anyone building agents

• Tool descriptions are a lever as strong as the model. With the model fixed, rewriting skills and MCP tool descriptions moved the score by roughly 40 points. If your agent picks the wrong tool or misuses arguments, fix the description and schema first. Our guides to agent skills and MCP explain the mechanics, and the Agent Reflex paper on noisy MCP tools shows the same failure class from the research side.
• Efficiency and accuracy rose together. Fewer turns and about 64% fewer output tokens mean lower cost per successful result, which is the metric the post recommends comparing.
• Grade the artifact, not the call. A successful tool call tells you nothing about whether the library is complete.
• Make cheating impossible to reward. The integrity gate is worth copying in any domain where the agent can reuse or fabricate results.
• It is a pattern, not a product. NVIDIA says the same method applies to EDA, CAE and other engineering domains: evaluate final artifacts, write a domain verifier, collect many rollouts, and optimize skills and tool interfaces.

Chip design has attracted other agent efforts, such as OpenAI's Synopsys GPT work, and the broader supply-side story is in our AI chip supply chain roundup.

What people are asking

Did a better model cause the jump? No. The post fixes the model and harness (Claude Opus 4.8 in Claude Code) and changes only skills, tool descriptions and schemas. That makes it a clean ablation of the scaffolding around the model, which is where many teams under-invest.

Why did turns drop so much? The authors point to simplified overlapping file and shell interfaces and explicit JSON schemas. When two tools can do the same job, an agent wastes turns choosing, retrying and recovering. Fewer, clearer tools mean fewer detours.

Does this need NVIDIA hardware? The post does not say the evaluation needs any specific GPU; it describes an environment, verifier and rollouts. The Solido flow it evaluates uses fast simulation, and NVIDIA ties faster simulation to faster agent iterations, but the verifier pattern itself is hardware-agnostic.

Can I copy this without chip-design expertise? The shape transfers: pick one task with a reference output, define gates for completeness, validity and no cheating, and score the rest by tolerance. The domain knowledge goes into the verifier, which is why the post stresses writing a domain-specific grader.

Is 95% good enough for production? That depends on the cost of the other 5%. In chip design, a wrong timing value can be expensive, which is why the authors keep a human in the loop and why the final-artifact verifier matters more than the score itself.

What to treat carefully

• It is a vendor post. The authors are from NVIDIA and Siemens, and the product being showcased includes NVIDIA's own library.
• Development subset. Two cells across 36 corners, not the full 229, were used for tuning. The post does not give held-out results on the full set, other models, or variance across rollouts.
• Human in the loop. The loop was driven by a coding agent with a human in the loop, so the 95% is not autonomous operation.
• Fixed model. Results for Claude Opus 4.8 may not transfer to other models, which the post does not test.
• No public numbers on cost. Wall-clock time per run is given (1.5 to 2 hours) but not dollar cost.

How to try it

NVIDIA points to the NeMo Gym Quickstart and suggests applying the approach to one representative task: define a valid final artifact, build a domain-specific verifier, inspect rollout failures, refine skills and tool descriptions, and rerun under consistent conditions. A starting template for your own agent:

Task: one representative job with a known-good reference artifact
Verifier: gates (coverage, validity, no regressions, integrity) x fraction within tolerance
Rollouts: 20+ from a clean state, same model and harness
Fix: ambiguous skill text, overlapping tools, missing JSON schemas
Compare: verified score, turns, output tokens per success

Related reading

• Build-A-Chip-Bench: best agent scores 16.6% on end-to-end chip design

• What are agent skills? A complete guide

• What is MCP?

• Agent Reflex: reflective tool agents and noisy MCP

• FrontierCode 1.1 and maintainer-merge benchmarks

• OpenAI and Synopsys: GPT for chip design

• NVIDIA Run:ai open source KAI scheduler

• Official: NVIDIA Technical Blog post

Details reflect NVIDIA's October 14, 2026 post and may change.