Nvidia announced a paired model-and-router approach aimed at reducing cost and maintaining performance for always-on agentic AI: Nemotron 3.5 Lightning, a 30-billion-parameter open mixture-of-experts model built for high-volume specialized agent tasks, and NeMo Switchyard, an open-source routing library that selects the best model for each step in an agent workflow. The company says the combination lets operators route simple steps to cheaper, faster models while preserving frontier-level task completion.
Why routing matters and what Switchyard does
Enterprises that run always-on agents face a tradeoff: sending every task to a frontier model is expensive, while building custom routing logic to send easy tasks to cheaper models becomes a separate engineering project that must be maintained. NeMo Switchyard aims to automate per-step routing by using live signals — agent state, tool outputs, errors and predicted token verbosity — to pick the most appropriate model for each step.
Nvidia stresses that pairing an open model with an open router under a single licensing and integration strategy is central: a model alone does not fully address cost, and a router alone has limited value unless there are efficient models to route to. Kari Briski, vice president of generative AI at Nvidia, summarized the approach as matching the right model to each workflow step.
Switchyard supports multiple routing strategies, including random routing, agent-state routing and classifier-based routing. Cost considerations are baked into routing decisions: Switchyard can estimate how many tokens a model is likely to emit for a task and prefer cheaper options accordingly.
Integration ecosystem
Rather than replacing existing routing services, Switchyard integrates with them. Nvidia listed two groups of partners: agent frameworks that call Switchyard directly (Cognition, LangChain and Nous Research) and LLM gateways that have added Switchyard support to their products (Kong, LiteLLM and OpenRouter). Kong ships Switchyard natively inside Kong AI Gateway, and Nvidia says OpenRouter and LiteLLM have also integrated the routing algorithm.
This partner-first approach is meant to minimize additional integration work for developers by enabling Switchyard to plug into tooling they already use.
Early test results
Nvidia shared results from nine companies testing Switchyard, with several specific claims:
- LangChain reported a 74% cost reduction across 145 multi-turn Deep Agents tasks by routing only 7% of calls to a frontier model, at a 6% accuracy tradeoff.
- Ramp said it matched a frontier model's performance on Ramp SWE-Bench while cutting costs 58% and runtime 33%.
- Cognition integrated Switchyard's staged router into Devin Desktop for internal use and reported near-frontier performance on FrontierCode Main while reducing mean cost 28% compared with routing everything to a single frontier model.
These cases indicate that intelligent routing can deliver substantial cost savings with limited accuracy tradeoffs in practice.
Nemotron 3.5 Lightning: architecture and targeted performance
Nemotron 3.5 Lightning is an open model intended as a fast, inexpensive option inside a routing setup rather than a general-purpose leader. It continues the hybrid Mamba-Transformer latent mixture-of-experts architecture Nvidia introduced with the Nemotron 3 family and used for Nemotron 3 Super, which Nvidia uses as Lightning's baseline in some comparisons.
On the Artificial Analysis Intelligence Index (a nine-evaluation general capability benchmark), Lightning scores 24, tied with gpt-oss-120b and trailing models such as Nemotron 3 Super, Gemma 4 31B, Claude 4.5 Haiku and Mistral Medium 3.5, all of which scored higher. Nvidia does not present Lightning as a general-intelligence leader in its size class.
The narrower claim is speed-to-accuracy: per PinchBench data Nvidia provided, Lightning completes agentic tasks roughly 30% faster than Qwen3.6-35B at comparable accuracy, and outperforms Gemma 4 26B on accuracy at similar completion time in PinchBench, a real-world agent benchmark covering coding, research and file management. Nvidia emphasizes this is a tradeoff focused on latency and cost rather than an overall capability win.
Post-training fine-tuning is where Nvidia says larger practical gains appear. Nvidia shared before-and-after figures from early-access partners — CrowdStrike, CodeRabbit, Harvey and Trajectory, and Lila Sciences — showing task-specific improvements versus baselines such as Nemotron 3 Super and Opus 4.x models. For example, Nvidia reports that CodeRabbit used a standard NeMo Auto model recipe trained for one epoch and obtained a working routing agent in about two hours at an estimated $85 cost in the provided case.
What this means for enterprises
The release highlights several trends enterprises should watch:
- Routing decisions are becoming dynamic: instead of fixed model assignments, pipelines are shifting to per-step routing based on live signals like agent state and token cost.
- Open source is a cost lever at two layers: model weights and routing code. Nvidia argues that controlling both under an open license can produce more meaningful cost reductions than offering cheaper weights alone.
- The competitive axis shifts from single-model performance to system-level effectiveness: as routing libraries mature, differentiation will increasingly depend on how well a routing layer pairs models to production tasks, which is harder to benchmark or market than raw model scores.
Nvidia's Nemotron 3.5 Lightning and NeMo Switchyard therefore represent a strategic bet that owning and open-sourcing both the model and routing layers will move the cost needle for agentic AI. Early partner results are promising, but broader market impact will depend on competitor responses and real-world adoption across varied workloads.
