NVIDIA has launched NeMo Switchyard, an open-source model-routing system designed to help AI agents choose the right model for each step instead of sending every request to one expensive frontier model.
Released on August 11, 2026, Switchyard arrived alongside NVIDIA Nemotron 3.5 Lightning, an open model built for high-volume agent work. NVIDIA’s strategy is straightforward: use faster or cheaper models for routine tasks, then call a stronger model only when deeper reasoning is needed.
That approach could matter as AI agents make more model calls across coding, research, customer support and enterprise automation.
In a LangChain benchmark, routing between Nemotron 3.5 Lightning and Claude Opus 4.8 reduced cost by 74% compared with using Opus alone, although accuracy also fell by roughly six percentage points.
The result makes Switchyard an interesting cost-efficiency story, but not a simple case of getting cheaper AI without tradeoffs.
What Is NVIDIA NeMo Switchyard?
NVIDIA NeMo Switchyard is an open-source AI model router that directs requests across different models based on capability, cost, latency, workflow stage and runtime conditions.
Think of it as a traffic controller between an AI agent and the models available to that agent.
A traditional AI application may choose one model and send most or all requests to it. Switchyard adds another layer that can determine which available model should handle the current request or stage.
NVIDIA’s technical guide to NeMo Switchyard explains that it supports multiple routing strategies across specialized and frontier models.
NeMo Switchyard at a Glance
| Feature | Details |
| Developer | NVIDIA |
| Type | Open-source AI model router |
| Main purpose | Route AI agent tasks to suitable models |
| Key signals | Capability, cost, latency and workflow state |
| Model approach | Specialized, local and frontier models |
| Related release | Nemotron 3.5 Lightning |
| Main use case | Multi-model and agentic AI workflows |
NVIDIA has also published the code in its Switchyard open-source repository. The project functions as a Rust proxy and library for routing LLM traffic across providers while supporting different API formats and routing policies.
Why Do AI Agents Need Model Routing?
AI agents can make dozens or even hundreds of model calls while working toward a single goal.
A coding agent, for example, may need to:
- Understand a request
- Inspect files
- Plan a solution
- Generate code
- Run tools
- Read errors
- Revise its output
- Validate the result
Not every step needs the most powerful model available.
A simple extraction or formatting task may work well on a smaller model. Complex debugging or planning may require a more capable frontier model.
Model routing tries to match the difficulty of each task with the right level of performance and cost.
This fits a wider shift toward more efficient AI systems. Compact models are also moving closer to devices, as seen with Liquid AI’s on-device AI agents, where smaller models are being designed for environments that cannot rely on huge cloud systems for every operation.
How NVIDIA NeMo Switchyard Routes AI Models
Switchyard does not depend on only one routing method.
NVIDIA describes both tuning-free and trainable approaches that can make decisions according to different signals.
LLM Classifier Router
An LLM classifier examines a request and selects a model from the available pool.
A straightforward task could go to a lower-cost model, while a request involving deeper analysis could be directed to a stronger model.
Stage Router
The stage router considers where an AI agent currently is within a workflow.
Example: Routine Work vs Difficult Recovery
An agent successfully editing files could continue using a cheaper model.
If it reaches repeated errors, enters an exploration stage or needs deeper reasoning, the router can direct the next step to a more capable model.
This matters because the difficulty of an agent task can change while the workflow is still running.
Escalation Router
Escalation routing starts with a lower-cost model and moves to a stronger one only when necessary.
A simplified process looks like this:
- A lower-cost model attempts the task.
- The system evaluates whether progress is being made.
- Failure, loops or uncertainty can trigger escalation.
- A more capable model handles the difficult step.
This approach can help reduce expensive frontier-model calls without removing access to them completely.
Prefill Router
NVIDIA also describes a trainable prefill router.
It uses model signals to estimate which candidate model is most likely to complete a request successfully, allowing routing decisions to become more predictive rather than depending only on fixed rules.
LangChain Reports 74% Lower Cost in a 145-Task Test
One of the most important public evaluations of Switchyard comes from LangChain.
In its NeMo Switchyard routing benchmark, LangChain tested 145 multi-turn agent tasks using Claude Opus 4.8, Nemotron 3.5 Lightning and a routed combination of the two.
Switchyard Cost and Accuracy Results
| Configuration | Accuracy | Cost per Run | Cost per Completed Task |
| Claude Opus 4.8 | 86.0% | $11.45 | $0.092 |
| Opus 4.8 + Nemotron 3.5 Lightning | 80.0% | $3.00 | $0.026 |
| Nemotron 3.5 Lightning only | 77.7% | $0.72 | $0.006 |
The routed setup cost about 74% less than Claude Opus 4.8 alone.
However, there was a clear tradeoff.
Accuracy decreased from 86% with Opus alone to 80% with the routed configuration.
Only 7% of Calls Used Claude Opus 4.8
The routing distribution reveals why the savings were so large.
LangChain reported:
- 93% of model calls went to Nemotron 3.5 Lightning
- 7% went to Claude Opus 4.8
The frontier model therefore handled only a small portion of the total calls.
A Few Frontier Calls Still Drove Most of the Cost
There is another important detail.
Although Opus handled only 7% of calls, it accounted for 68.4% of routed spending.
Nemotron 3.5 Lightning handled 93% of calls while representing only 10.4% of the spend.
Routing Itself Also Costs Money
The judge model used in the routing process accounted for another 21.2% of spending.
That means routing is not free.
Companies evaluating model routers need to measure the cost of the routing logic alongside the cost of the models producing responses.
Why the 74% Cost-Saving Figure Needs Context
It would be misleading to say that NeMo Switchyard automatically cuts every company’s AI bill by 74%.
The LangChain result came from a specific benchmark, model combination and routing setup.
Actual savings will depend on factors such as:
- Model pricing
- Task complexity
- Escalation frequency
- Routing overhead
- Required accuracy
- Number of agent steps
- Local versus API-based deployment
There is also an important comparison with Nemotron Lightning alone.
Nemotron 3.5 Lightning reached 77.7% accuracy at $0.72 per run, while the routed system reached 80% at $3.00.
That means a routing system is not automatically the cheapest option.
For workloads where minimum cost matters more than a small improvement in accuracy, using the cheaper model alone may sometimes make more sense.
LangChain also cautioned that teams should test the tradeoff against their own workloads rather than assuming one benchmark will apply everywhere.
What Is NVIDIA Nemotron 3.5 Lightning?
Nemotron 3.5 Lightning is the fast execution model at the center of NVIDIA’s multi-model strategy.
NVIDIA describes it as an open model designed for high-volume workloads used by long-running and always-on AI agents.
The model has roughly:
- 30 billion total parameters
- About 3 billion active parameters
- A hybrid Mamba-2 and mixture-of-experts architecture
- Support for high-volume agent workloads
The practical goal is more important than the architecture alone.
Nemotron Lightning is intended to perform large volumes of useful work efficiently while more expensive models remain available for the hardest parts of a workflow.
What Tasks Could a Faster Model Handle?
Depending on the application, these tasks could include:
- Tool calls
- Information extraction
- Data formatting
- Result validation
- Retrieval
- Routine coding operations
- Repetitive sub-agent work
NVIDIA says Nemotron 3.5 Lightning can deliver up to 4x the output speed of similar-sized models in its own testing. Because this is a vendor-reported performance claim, organizations should validate the result against their own infrastructure and workload.
NVIDIA’s broader focus on agentic systems can also be seen in its expanding AI research partnerships, including the South Korea AI strategy involving NVIDIA and KAIST.
Why Model Routing Could Matter for Enterprise AI
For years, one of the biggest AI questions has been:
Which model is best?
Model routing changes the question to:
Which model is best for this specific step?
Enterprises increasingly have access to different types of models:
- Large frontier models
- Fast mid-sized models
- Small local models
- Coding-focused models
- Retrieval models
- Domain-specific models
A smart routing layer could combine these options instead of asking one model to handle everything.
This approach may help businesses reduce unnecessary frontier-model use while retaining access to stronger models for difficult work.
It also gives companies more flexibility to combine cloud APIs with local or specialized systems.
The same push for better cost and infrastructure control is visible at the hardware level. For example, Anthropic’s reported AI chip strategy highlights how AI companies are exploring ways to gain more control over the computing costs behind training and inference.
What Does Switchyard Mean for Claude and Other Frontier Models?
Switchyard does not make frontier models unnecessary.
LangChain’s test actually showed that Claude Opus 4.8 achieved the highest accuracy of the three configurations.
What may change is how often an expensive frontier model needs to be used.
Instead of sending every step to a premium model, developers can reserve it for tasks where additional reasoning ability produces meaningful value.
That issue becomes increasingly important as providers compete on both intelligence and price. The changing economics can also be seen in Claude Opus 5 pricing and enterprise benchmarks, where model cost is increasingly part of the enterprise deployment decision.
Main Tradeoffs Enterprises Should Consider
Switchyard creates an attractive cost story, but multi-model routing introduces new challenges.
Accuracy Can Fall
LangChain’s routed configuration scored six percentage points below Opus alone.
That difference may be acceptable for routine automation but more important in applications where mistakes carry higher costs.
Routing Adds Operational Complexity
Teams need visibility into:
- Which model handled each request
- Why that model was selected
- Whether fallback rules worked
- How failures are diagnosed
- How quality changes between models
Governance Can Become More Complicated
Multi-model systems may require stronger controls around:
- Data privacy
- Logging
- Model attribution
- Audit trails
- Provider access
- Compliance requirements
Benchmarks May Not Match Production
A 145-task evaluation cannot represent every production workload.
Coding agents, customer-service systems, financial workflows and research agents may behave very differently.
Businesses should therefore test model routing using workloads that closely resemble their actual production environment.
Frequently Asked Questions
What is NVIDIA NeMo Switchyard?
NVIDIA NeMo Switchyard is an open-source model-routing system that helps AI agents choose between different models according to capability, cost, latency, workflow stage and other runtime signals.
How Does NeMo Switchyard Lower AI Agent Costs?
It can send routine tasks to lower-cost models while reserving expensive frontier models for steps that need stronger reasoning.
Did NeMo Switchyard Really Cut Costs by 74%?
LangChain reported a 74% cost reduction in its specific 145-task benchmark compared with using Claude Opus 4.8 alone. The routed setup also had lower accuracy, so the figure should not be treated as a universal saving.
Can Switchyard Route Claude and Other Third-Party Models?
Yes. Switchyard is built to route LLM traffic across different providers rather than limiting developers to NVIDIA models.
Is NVIDIA NeMo Switchyard Open Source?
Yes. NVIDIA has publicly released the Switchyard project, including its routing proxy and library.
Final Takeaway
NVIDIA NeMo Switchyard tackles a practical problem in agentic AI: how to use expensive intelligence only when it is actually needed.
LangChain’s early benchmark shows why the idea is attractive. Most calls were handled by the cheaper Nemotron 3.5 Lightning model, while a small percentage of Claude Opus calls preserved access to stronger reasoning.
But the tradeoff matters just as much as the savings.
Routing can introduce accuracy loss, additional orchestration costs and more operational complexity.
For enterprises, the real question is not whether Switchyard can produce one impressive cost-saving number. It is whether dynamic model routing can consistently deliver the right balance of quality, speed and cost on real production workloads.
