As AI agents become more capable, the cost of running them is becoming a bigger challenge for developers and enterprises. An agent may need dozens or even hundreds of model calls to complete a single complex workflow. Sending every one of those requests to a powerful frontier model can deliver strong results, but it can also make large-scale deployment expensive.
NVIDIA is now targeting that problem with NeMo Switchyard, an open-source model-routing library designed to decide which AI model should handle different parts of an agentic workflow.
At the same time, NVIDIA has introduced Nemotron 3.5 Lightning, a 30-billion-parameter open mixture-of-experts model designed for high-volume agentic workloads. The company says the model can deliver up to four times faster output and 30% faster agentic task completion than comparable models in its class.
The bigger development, however, may be Switchyard. NVIDIA says its internal testing showed that routing tasks between models could reduce task-completion costs to nearly one-third of using a single frontier model alone. That represents a potential cost reduction of roughly two-thirds, although the figure comes from NVIDIA’s own benchmark and should not be treated as a universal production result.
NVIDIA Introduces NeMo Switchyard for Agentic AI
NeMo Switchyard is designed around a simple idea: not every step performed by an AI agent requires the same level of intelligence.
A complex coding problem, for example, may require a powerful reasoning model. A later step that formats information, calls a tool or performs a routine operation may not need the same capability.
Instead of forcing one model to perform every operation, Switchyard can route different requests to models that are better suited to each task. NVIDIA describes the system as an open-source routing library that can optimize AI workflows around quality, latency and cost.
This approach matters because agentic AI is fundamentally different from a single chatbot response. An autonomous agent can plan, reason, call tools, inspect results, recover from errors and continue working through multiple stages.
Each additional model call can add inference cost and latency. A routing layer therefore gives developers another way to control the economics of these workflows.
NVIDIA’s strategy also fits into its broader push toward open AI infrastructure. The company says the new tools are intended to give developers more control over where AI runs, how it is deployed and how workflows evolve across PCs, workstations, data centers and the cloud.
How NVIDIA NeMo Switchyard Routes AI Models
The easiest way to understand Switchyard is to think of it as a dispatcher for AI models.
An agent receives a request and begins working through its task. Instead of automatically sending every step to the same model, the routing layer can determine which model is appropriate for the current stage.
A simplified workflow looks like this:
AI agent → Switchyard → appropriate model → result → next agent step
For routine operations, the system can favor an efficient model. When the workflow reaches a difficult reasoning problem, it can use a more capable model.
NVIDIA’s technical documentation describes multiple routing approaches, including random routing, classifier-based routing, session affinity and stage-based routing. The stage-router approach is particularly relevant to agentic workflows because it can use the current stage of a task to determine which model should handle it.
For example, an agent could use an efficient model for routine execution and reserve a more capable model for exploration, difficult reasoning or error recovery.
That creates an important shift in how AI systems can be designed. Instead of asking, “Which single model is best?”, developers can ask, “Which model is best for each step?”
Why Model Routing Can Reduce AI Inference Costs
The economics behind Switchyard are relatively straightforward.
Frontier AI models are expensive because they provide advanced reasoning and broad capabilities. But an agent does not necessarily need those capabilities during every step of a workflow.
Consider a coding agent working on a software project. The hardest part may involve understanding an unfamiliar codebase and deciding how to fix a complex bug. Later, the same agent might need to execute a tool call, inspect a straightforward output or make a routine change.
Using the most expensive model for both stages can waste computing resources.
Model routing attempts to reserve expensive inference for situations where it provides meaningful value.
This is particularly important for enterprise AI because agents can operate continuously and generate large numbers of model calls. Even a modest saving per request can become significant when multiplied across millions of interactions.
NVIDIA says its internal benchmark showed Switchyard could achieve frontier-level accuracy while reducing task-completion cost to nearly one-third compared with using Opus 4.8 alone. The important wording is “to nearly one-third”, which means approximately a 67% reduction rather than a 33% reduction.
However, that result should be viewed as a benchmark rather than a guaranteed saving for every deployment.
Partner Results Show Different Levels of Cost Savings
NVIDIA’s announcement also includes results from companies testing Switchyard in different AI workflows.
Ramp reported a 58% reduction in cost and a 33% reduction in runtime.
LangChain reported a 74% lower cost across 145 multi-turn Deep Agents tasks by routing only 7% of calls to a frontier model. NVIDIA says this result came with a 6% accuracy trade-off.
Cognition reported a 28% reduction in mean cost, while Classmethod reported a 27% reduction while maintaining quality.
Other results focus on efficiency and latency. NVIDIA says Boomi achieved 100% domain-routing accuracy and 21% lower latency on later turns, while Cadence reported a 9.9% efficiency improvement.
These results are useful because they demonstrate that the economics of model routing can vary significantly depending on the workflow.
A company prioritizing maximum accuracy may route more requests to a frontier model. Another organization focused on high-volume automation may accept a small quality trade-off to achieve much larger cost savings.
That makes model routing less about finding one universal configuration and more about controlling the quality-cost-latency trade-off.
Nemotron 3.5 Lightning Is Built for the Efficient Side of the System
NeMo Switchyard is the routing layer, while Nemotron 3.5 Lightning provides another model option within that broader ecosystem.
NVIDIA describes Nemotron 3.5 Lightning as a 30-billion-parameter model with approximately 3 billion active parameters per token. Its mixture-of-experts architecture is designed to provide efficient performance for high-volume agentic workloads.
The distinction is important.
Switchyard does not simply make one model faster. It changes how an AI system decides which model should perform a particular task.
Nemotron 3.5 Lightning, meanwhile, gives developers an efficient model that can potentially handle many of the routine or high-volume operations within those workflows.
NVIDIA says Lightning can run across different environments, including local systems and NVIDIA infrastructure. That flexibility could be important for enterprises that need greater control over proprietary information, deployment and infrastructure costs.
For businesses exploring AI tools and enterprise automation, this type of architecture could eventually become more important than simply comparing individual models by benchmark score.
Why AI Agents Need More Than One Model
The growth of agentic AI changes the economics of model selection.
A traditional chatbot may answer a user’s question with one or a few model calls. An AI agent can perform a much longer chain of actions.
It may:
- Understand a request
- Create a plan
- Search for information
- Call external tools
- Analyze results
- Write or modify code
- Check its own work
- Recover from errors
- Complete the final action
Those steps do not necessarily require identical reasoning capabilities.
Using one expensive model throughout the workflow can therefore become inefficient.
A multi-model architecture allows developers to assign different jobs to different models. This is similar to how conventional software systems use specialized components rather than relying on a single program to perform every operation.
The difference is that AI model routing can make those decisions dynamically.
That could make model routing an increasingly important layer of AI agent infrastructure, particularly as organizations move from experimental assistants toward autonomous workflows.
Where Switchyard Could Matter Most
The biggest opportunity for model routing is likely to appear in workflows with high model-call volume.
Coding agents
Coding agents can alternate between difficult reasoning and routine execution. A capable model can handle complex debugging while a smaller model handles straightforward operations.
Enterprise support
Customer-service agents may need powerful reasoning for unusual cases but can use efficient models for routine requests, classification and workflow actions.
Cybersecurity
Security agents can process large numbers of alerts and reserve stronger models for incidents that require deeper investigation.
Business automation
Agents handling invoices, reports, documents and internal workflows may perform many repetitive steps where frontier-level reasoning is unnecessary.
Research and analysis
Research agents can use different models for retrieval, summarization, classification and complex synthesis.
The core opportunity is the same in each case: match model capability to task difficulty instead of paying the highest inference price for every operation.
NVIDIA Is Building an Ecosystem Around Model Routing
NVIDIA is also positioning Switchyard as part of a broader AI infrastructure ecosystem.
The company says LangChain has tested the technology with Deep Agents, while LiteLLM is adding Switchyard as a plug-in to its proxy layer. NVIDIA also highlights integration with Kong AI Gateway.
That matters because developers are unlikely to rebuild their entire AI stack simply to adopt a routing system.
Integration with existing gateways, frameworks and model infrastructure can make routing easier to introduce into production systems.
This also moves Switchyard beyond being just another NVIDIA model announcement. If routing becomes integrated into the infrastructure developers already use, it could become a more fundamental part of how enterprises manage AI inference.
The Limitations of AI Model Routing
Despite the potential savings, model routing is not a free optimization.
The first challenge is accuracy.
If a router sends a difficult request to a model that is too weak, the overall agent may produce a worse result. NVIDIA’s own LangChain result illustrates this trade-off, with a reported 74% cost reduction alongside a 6% accuracy trade-off.
The second challenge is routing overhead.
A system needs enough information to determine which model should handle a request. If the routing process itself adds significant latency or inference cost, some of the expected benefit can disappear.
The third challenge is workload variability.
A routing configuration that works well for a coding agent may not produce the same results for customer support or scientific research.
For that reason, enterprises should evaluate routing against their own workloads rather than assuming NVIDIA’s benchmark results will translate directly to production.
The most useful metrics are likely to include:
- Cost per completed task
- Latency
- Accuracy
- Number of model calls
- Token consumption
- Error rate
- Successful task completion
NVIDIA’s Switchyard documentation also emphasizes monitoring model calls, tokens, latency and cost, highlighting the importance of observability when optimizing multi-model workflows.
What NVIDIA Switchyard Means for the Future of AI
The significance of NeMo Switchyard extends beyond one routing library.
The AI industry has spent much of the last few years competing to build increasingly capable models. But as AI agents become more autonomous, inference economics are becoming just as important as raw model capability.
A highly intelligent model is valuable, but using it for every small operation may not be economically sustainable.
Model routing offers another approach.
Instead of building an agent around one model, developers can build a system in which several models cooperate. Strong models handle difficult decisions, efficient models process routine operations and a routing layer determines where each task should go.
That could change the way businesses evaluate AI systems.
The question may increasingly become not simply which model is smartest, but which combination of models can complete a task most efficiently without sacrificing the required quality.
NVIDIA’s latest release is therefore as much about AI infrastructure as it is about another open model.
NVIDIA NeMo Switchyard Could Reshape Agentic AI Economics
NeMo Switchyard addresses one of the most practical problems facing agentic AI: the cost of repeatedly using powerful models across long workflows.
NVIDIA’s internal benchmark suggests that intelligent routing can reduce task-completion costs to nearly one-third of a single-frontier-model approach. Partner results show that real-world savings can vary substantially depending on the workload, quality requirements and routing strategy.
Nemotron 3.5 Lightning strengthens that strategy by providing an efficient model designed for high-volume agentic workloads.
The broader trend is clear. As AI moves from simple chat interactions toward autonomous systems, model routing, inference optimization and cost control are becoming core parts of the AI stack.
Switchyard’s long-term importance may therefore depend less on one benchmark and more on whether developers adopt the idea behind it: use the strongest model when it matters, and a more efficient model when it doesn’t.
