NVIDIA's NeMo Switchyard is an open-source, provider-agnostic SDK and orchestration layer that enables a "system-of-models" approach for AI agents. Instead of sending every request to a single model, NeMo Switchyard routes individual agent workflow steps to the model best suited for capability, latency and cost. The aim is to improve efficiency and control in real AI workflows by combining specialized and frontier models dynamically.
How routing decisions are made and what signals are used
At runtime, a router evaluates each request and its available context, then directs the work to an appropriate model from a configured pool. Effective routing uses signals from three primary areas:
- Model capabilities: which model(s) can correctly solve the task.
- Model cost profile: latency and monetary cost for running or accessing each model.
- Infrastructure: system-level signals such as pricing, latency, load, and error conditions that enable reliable handoffs.
Routers can inspect the request itself (topic classification, embeddings, feature extraction), model state (logprobs, residual streams, attention matrices, agent traces), and system metrics (current latency, pricing, load). Routers must also decide when and where to evaluate signals — for instance, routing whole requests to a model versus routing each step in a multi-turn agent interaction.
Architecture and tooling
The provider-agnostic SDK behind NeMo Switchyard (switchyard-lib) represents requests, defines available models, and manages calls to a chosen model. Each model target is given a semantic name; the client maps that name to a provider endpoint and model ID. This separation keeps routing logic independent of a particular provider or endpoint.
NeMo Switchyard can persist routing state across an agent session when policies or later decisions require historical context (e.g., previous tool outputs or affinity decisions). It can also operate statelessly when history is unnecessary. Because model deployments change, this design lets teams update models, move endpoints, or switch providers without modifying routing logic.
A NeMo Switchyard server implementation acts as a reference for exposing routing through common APIs, simulating an LLM gateway. It accepts OpenAI, Anthropic, and Responses API requests, translates them into NeMo Switchyard's internal format, and returns the expected responses. The server also records the selected model, decision rationale, token usage, latency, and outcomes for inspection.
Routing approaches: tuning-free and tunable routers
NeMo Switchyard supports both tuning-free (heuristic) routers and tunable (learned) routers.
Tuning-free routers
-
LLM classifier: uses an LLM as a judge to select a candidate model and maintains session affinity to avoid repeated reclassification over a task arc. This suits headless or domain-specific systems (e.g., routing coding, math, or healthcare tasks to targeted models).
-
Stage router: applicable for coding agents that pass through stages (exploration, error recovery, mechanical implementation). The stage router inspects recent tool activity to decide whether a turn needs a higher-capability model (e.g., on severe errors or prolonged exploration) or a more efficient model for steady edits once tests pass.
-
Escalation router: begins conversations with a lower-cost model and uses an LLM judge to monitor progress turn by turn. If sustained difficulty is detected (errors, loops, drift), the session escalates to a more capable model. This is designed for multi-turn agent workloads where routine work can be handled by smaller models but harder cases need escalation.
Tunable routers
Tunable routers replace fixed heuristics with signals learned from workload data, predicting how likely each candidate model is to answer correctly.
- Prefill router: during training, it extracts the LLM residual stream to estimate query complexity. A shared-trunk MLP learns mappings from residual signals to accuracy labels for each model in the pool. At inference, the prefill state is input to the router, which predicts the probability each model will succeed. A policy then trades predicted accuracy against cost, latency, or other constraints to select the best model for a request.
Benchmarks and partner deployments
LangChain benchmark: using an internal deep agents evaluation suite of 145 multi-turn agentic tasks, routing between NVIDIA Nemotron 3.5 Lightning and Claude Opus 4.8 with the escalation router produced a 74% cost reduction compared with a frontier-only baseline across five runs. In that experiment, only 7% of calls were sent to the frontier model, at an observed ~6-point accuracy trade-off.
Cognition deployment: Cognition implemented the staged-routing approach in Devin Desktop and tested it with NVIDIA internal users. On the FrontierCode Main benchmark for production-grade coding tasks, routing between Opus 5 and Kimi K2.7 achieved 50.6% accuracy at a $3.11 mean cost — within 2.8 percentage points of Opus 5 accuracy and at approximately 28% lower mean cost.
These results demonstrate practical cost and efficiency gains from model-neutral, adaptive routing while preserving near-frontier performance in many cases.
Integrations and availability
NVIDIA is working with partners across agent tooling, model providers and enterprise applications to bring NeMo Switchyard into existing developer workflows. Integrations and collaborations include Cognition, Nous Research, Ramp, LangChain, LiteLLM, Kong, Classmethod, Boomi Agent Garden, as well as industry-specific work with Cadence and Siemens.
NeMo Switchyard is fully open source on GitHub; developers can use the SDK, test routing algorithms, and contribute custom approaches for their use cases.
Conclusion
As AI systems increasingly combine specialized and frontier models, routing becomes essential for selecting the right model for each task step while balancing quality, latency and cost. NeMo Switchyard provides a practical, provider-agnostic toolkit and reference server to help teams adopt model routing without rebuilding applications around each provider or model change.
