NVIDIA's NeMo Switchyard and the Rise of the Multi-Model AI Gateway
Good news! NVIDIA has open-sourced NeMo Switchyard, an LLM traffic proxy and routing library written in Rust. Switchyard proposes that not every request needs to call a top-tier model. By using Intelligent Model Routing to distribute traffic on-demand across multiple models, it can reduce call costs while maintaining effectiveness.
As AI development accelerates, the large model market is highly fragmented. Dozens of vendors like OpenAI, Anthropic, Google Gemini, DeepSeek, Zhipu GLM, and Tongyi Qianwen each have their own strengths, and development teams rarely rely on a single vendor. The chaos of key management, opaque costs, difficult vendor switching, and single-point-of-failure risks that come with running multiple models in parallel has evolved into a systemic engineering problem.
NeMo Switchyard chooses to solve the model selection and protocol translation problem with an open-source proxy and routing algorithm library. ServBay AI Gateway offers its own solution on a more complete scale, covering the full chain of channel access, traffic scheduling, virtual keys, model mapping, protocol conversion, and usage statistics.
This article starts from the technical design of NeMo Switchyard, sorts out the key capability dimensions involved in intelligent model routing, and then expands item by item on how ServBay AI Gateway implements these capabilities in real-world scenarios.
What did NVIDIA NeMo Switchyard do?
NeMo Switchyard is open-sourced under the Apache 2.0 license, positioned as a proxy and routing library for LLM traffic, and is currently in the pre-alpha stage. It breaks the problem of routing LLM requests into three independent technical modules.
Protocol Translation
Switchyard supports bidirectional conversion between three API formats: OpenAI Chat Completions, Anthropic Messages, and OpenAI Responses. The official documentation gives a typical usage example: point Claude Code (which natively uses the Anthropic format) to Switchyard, and then have Switchyard forward requests to open-source models deployed on vLLM or Ollama. Claude Code continues to send requests in its native API format, and Switchyard handles the format conversion in the middle, requiring no adaptation on the tool side.
Routing Algorithms
Switchyard has four built-in routing strategies:
Random: Distributes randomly among candidate models, suitable for A/B testing.
LLM Classifier: Uses an LLM to classify requests and routes them to corresponding models based on the classification result.
Stage Router: Grades requests based on complexity signals; simple requests go to lightweight models, complex requests go to flagship models.
Escalation Router: An upgraded mode of the LLM Classifier; requests with low confidence are automatically escalated to a higher-capability model.
Among these, the Stage Router is the most watched algorithm. In a set of benchmark tests, by adjusting the Stage Router's confidence threshold from 0.3 to 0.5, the call proportion of the flagship model (Claude Opus 4.8) dropped from 85% to 17%, with the lightweight model (GLM-5.2) taking over most of the traffic. Ultimately, the cost per task dropped by 43.7%, while the task completion rate remained stable.
Operational Metrics
Switchyard exposes metrics in Prometheus format, covering request counts, error rates, latency, token consumption, and the additional overhead of routing decisions.
However, the barrier to entry for Switchyard is not low. Installation requires a Rust compilation environment or the Python uv toolchain, configuration is done through TOML files, and there is currently no graphical management interface. The official documentation also explicitly marks it as experimental software and does not recommend direct use in production environments.
Looking at the Problems an AI Gateway Needs to Solve from Switchyard's Capability Framework
Although NeMo Switchyard's design focuses on routing and protocol translation, it also illustrates a point: a complete AI traffic management solution needs to cover at least the following capability dimensions:
Multi-vendor Channel Access: How many vendors can be connected, and whether custom backends are supported.
Traffic Scheduling: How requests are distributed among multiple channels, and how to automatically fall back during failures.
Credential Security: How API Keys are stored and managed, and how to prevent leaks.
Model Mapping: Whether the model identifier at the application layer can be decoupled from the actual backend model.
Protocol Conversion: How to mask the differences in API formats between different vendors.
Usage Statistics: Whether token consumption, cost spending, and performance data can be tracked.
Switchyard goes deep on protocol translation and routing algorithms but leaves aspects like credential management, usage statistics, and channel health management for the user to solve on their own. This stems from Switchyard's positioning as a routing library, not a complete gateway product.
ServBay AI Gateway, on the other hand, is positioned as a full-featured AI gateway, covering a complete implementation of the six dimensions mentioned above.
Channel Management: Full Coverage from Official APIs to Relay Stations
ServBay AI Gateway comes pre-configured with access templates for nearly 20 vendors, placing it in the tier with relatively broad coverage among similar products:
International Mainstream: OpenAI, Anthropic, Google Gemini, Azure OpenAI, AWS Bedrock, OpenRouter
Domestic Vendors: DeepSeek, Tongyi Qianwen, Zhipu GLM, Kimi (Moonshot AI), Doubao (Volcano Engine), Wenxin Yiyan, Hunyuan, MiniMax, 01.AI, StepFun
Open-Source Models: Ollama, LM Studio
Custom: Any third-party service compatible with the OpenAI protocol
The custom type deserves a separate note. Various API relay stations commonly used by developers can be connected to the gateway through the OpenAI Compatible channel type, as long as they provide an interface compatible with the OpenAI Chat Completions format. Simply fill in the relay station's Base URL and Key, exactly the same operation as connecting to an official API.
Multiple channel instances can be added for the same vendor. For example, DeepSeek can be configured with a direct official connection and two different relay stations simultaneously. The three channels jointly serve requests for the DeepSeek series models, forming a priority gradient and redundancy guarantee in combination with subsequent traffic scheduling rules.
Unlike Switchyard, which requires manually writing TOML configuration files, channel management in ServBay AI Gateway is done in a graphical interface. Select the vendor type, paste the API Key, save, and it's online—no need to touch the command line or configuration files throughout the process.
Traffic Scheduling: Priority, Failover, and Hot Switching
NeMo Switchyard's traffic scheduling focuses on which model to choose, using the Stage Router's complexity assessment or the LLM Classifier's classification judgment to select between models of different capability levels.
ServBay AI Gateway's traffic scheduling focuses on which pathway to take. When the same model is configured with multiple channels (official API, relay stations, subscription accounts in different regions, etc.), it determines which channel each request goes through and how to automatically switch when a channel has a problem.
Channel Priority
Each channel can be assigned a priority weight. The gateway preferentially sends requests to high-priority channels, for example, setting the official direct API as the highest priority, a relay station as a backup, and a second relay station as a fallback.
Automatic Failover
When a channel returns a 429 (rate limit), 500 (service error), or a response timeout, the gateway automatically retries the request on the next available channel. The gateway internally maintains channel health status (normal, degraded, unavailable). Channels with consecutive exceptions are temporarily removed from the available pool and automatically rejoin after recovery.
The entire switching process is completely transparent to downstream tools like Claude Code and Cursor.
Channel Hot Switching
Channels can be enabled, disabled, or have their priority adjusted at any time in the management interface. Changes take effect in real-time without needing to restart the gateway service or modify downstream application configurations.
A practical scenario: The Anthropic official API suddenly imposes a rate limit. In the management interface, raise the priority of a specific Anthropic relay station to the highest, click save, and all requests are immediately switched over. The entire operation takes a short time, and Claude Code is completely unaware.
Virtual Keys: Key Security and Multi-Project Isolation
Switchyard passes API Keys through environment variables and does not provide an independent credential management mechanism. In scenarios with multiple parallel projects, how to securely store keys and isolate usage by project needs to be solved independently.
ServBay AI Gateway's Virtual Key is designed to solve this problem:
Real vendor API Keys are stored encrypted and will not appear in the configuration of any downstream tool.
The gateway issues virtual keys for different projects or tools. Each key can be bound to independent permissions (which models can be accessed, which channels can be used).
The call volume, token consumption, and cost for each virtual key are independently tracked.
# Project A's Claude Code
export ANTHROPIC_API_KEY="vk-project-a-xxxx"
export ANTHROPIC_BASE_URL="http://127.0.0.1:11580"
# Project B's Cursor
# API Key: vk-project-b-yyyy
# Base URL: http://127.0.0.1:11580
# Project C's self-developed application
# API Key: vk-project-c-zzzz
# Base URL: http://127.0.0.1:11580
The security advantage of virtual keys lies in fault isolation. If a virtual key is accidentally leaked into a Git repository or log file, simply revoke that key in the management interface and reissue a new one. The real vendor API Key is completely unaffected, and other projects are not disturbed.
For freelancers or studios serving multiple clients simultaneously, virtual keys also solve the cost accounting problem. Each client project uses an independent virtual key, and at the end of the month, the usage data for each key can be exported from the dashboard.
Model Mapping: Decoupling the Application Layer from Actual Models
In NeMo Switchyard's routing configuration, a Route registers a Model ID and maps it to different Targets through routing algorithms. In its benchmarks, the efficient tier uses GLM-5.2, and the frontier tier capacity uses Claude Opus 4.8. The Stage Router distributes requests between the two based on complexity signals.
ServBay AI Gateway's Model Mapping provides flexible name decoupling capabilities with a wider range of applications:
Model Mapping Examples:
claude-opus-5 → glm-5.2
gpt-4o → deepseek-v4-pro
my-default → claude-sonnet-4
team-fast → deepseek-v4-lite
team-strong → claude-opus-5
For example, if the upper-layer application specifies the use of claude-opus-5, the gateway automatically maps and replaces it with the underlying glm-5.2 during forwarding. The upper-layer application just initiates requests without changing a single line of code; the underlying model can be swapped at any time.
This enables:
Model Evaluation and A/B Comparison: When evaluating whether DeepSeek V4 Pro can replace GPT-4o, modify the mapping target at the gateway layer, run the same set of test cases against both models, and compare latency, output quality, and cost. This aligns with the A/B benchmarking scenario mentioned in Switchyard's documentation.
Cost Tiering: Switchyard's Stage Router implements request tiering through automated complexity assessment. If automatic judgment is not needed, a similar effect can be achieved manually in ServBay AI Gateway through model mapping, defining different model identifiers for different tasks and mapping them to models at different price points.
Team Naming Conventions: Define internal names like
team-fastandteam-strong, mapping them to specific vendor models. When the underlying model is replaced, only the mapping configuration needs to be modified; all project code automatically follows suit without needing individual modifications.
Protocol Conversion: Masking Vendor Format Differences
Protocol translation is one of the three core features of NeMo Switchyard. Switchyard supports interconversion between OpenAI Chat Completions, Anthropic Messages, and OpenAI Responses, allowing programming tools to call any backend in their native format.
ServBay AI Gateway has broader coverage in protocol conversion, fully supporting the underlying conversion of three mainstream formats: OpenAI, Anthropic, and Gemini.
In actual use, regardless of whether the upper-layer protocol is OpenAI, Anthropic, or Gemini, the lower-layer application just needs to call without thinking, completely eliminating the need to consider complex protocol adaptation:
Claude Code (native Anthropic Messages protocol) can seamlessly call the OpenAI-compatible interfaces of DeepSeek or Zhipu GLM through the gateway.
Tools that only support the OpenAI protocol can access the Anthropic Claude series models through the gateway.
Requests in the Gemini protocol can be routed to any OpenAI or Anthropic backend.
Developers and downstream applications do not need to care about which protocol the target vendor uses; the gateway automatically completes the format conversion of request and response bodies in the middle layer. Switching vendors does not require changing existing code or tool configurations.
Usage Statistics and Cost Analysis
NeMo Switchyard exposes Prometheus metrics, requiring external tools like Grafana to achieve visual monitoring.
ServBay AI Gateway builds statistical capabilities into a visual dashboard, ready to use out of the box:
| Metric Category | Specific Content |
|---|---|
| Request Dimension | Total requests, success rate, error classification (routing success, failover, failure) |
| Token Dimension | Input tokens, output tokens, total consumption |
| Cost Dimension | Spending tracked by model, by channel, and by virtual key |
| Performance Dimension | Average response latency, Time to First Token (TTFT) |
| Multimodal Dimension | Number of image generations, voice input and output consumption |
Data supports multi-dimensional grouping, observing trends by date, comparing consumption by model, and tracking project-level spending by virtual key. For scenarios requiring budget control, budget caps and alert thresholds can be set, automatically triggering notifications when spending approaches the limit.
This data helps developers answer a specific set of questions: How much was spent on the Anthropic API last month? How many more tokens did Project B use compared to Project A? Which model has the lowest Time to First Token latency? How many times was the relay station channel triggered by failover?
Capability Dimension Comparison: NeMo Switchyard vs. ServBay AI Gateway
| Capability Dimension | NeMo Switchyard | ServBay AI Gateway |
|---|---|---|
| Protocol Conversion | OpenAI / Anthropic / Responses interconversion | OpenAI / Anthropic / Gemini interconversion |
| Routing Algorithm | Random / LLM Classifier / Stage Router / Escalation | Channel priority + automatic failover + hot switching |
| Credential Management | Environment variables | Encrypted storage + virtual key isolation |
| Model Mapping | Route → Target configuration | Direct mapping configuration in GUI |
| Usage Statistics | Prometheus metrics (requires external tools) | Built-in visual dashboard |
| Vendor Coverage | Manual TOML configuration required | Nearly 20 pre-configured + custom relay stations |
| Operation Method | Command line + TOML config files | Graphical interface |
| Maturity | Pre-alpha (experimental software) | Released |
The focus of the two is different. Switchyard's focus is on the depth of routing algorithms; the Stage Router's complexity assessment and the LLM Classifier's automatic classification are capabilities that have no direct equivalent in ServBay AI Gateway. Conversely, ServBay AI Gateway provides more complete engineering coverage in dimensions such as the breadth of channel management (nearly 20 pre-configured vendors and relay stations), credential security (virtual keys), and visual statistics.
The two are not mutually exclusive choices. Teams with deep customization needs for routing algorithms can use Switchyard's routing capabilities while simultaneously using ServBay AI Gateway for infrastructure-level tasks like channel management, key security, and cost statistics.
Typical Use Cases
Scenario 1: Multiple Tools Sharing Multi-Vendor APIs
In daily development, Claude Code, Cursor, and Gemini CLI are used simultaneously. Without a gateway, the three tools need to be configured with API Keys for each vendor separately, with keys scattered in three places. After connecting to ServBay AI Gateway, all three tools point to the same endpoint, each using an independent virtual key. Real keys are centrally managed, usage is tracked separately, and switching vendors only requires an operation on the gateway side.
Scenario 2: Automatic Fallback for Vendor Failures
Use the DeepSeek API as the primary channel, while also configuring a DeepSeek relay station and an OpenAI-compatible channel as backups. When the official DeepSeek API encounters rate limiting or maintenance, the gateway automatically switches requests to the relay station channel, and development work is not interrupted.
Scenario 3: Model Replacement Evaluation
A team is evaluating replacing GPT-4o with DeepSeek V4 Pro to reduce call costs. In the gateway, map gpt-4o to deepseek-v4-pro, run the same code and test cases for a week, compare the differences in latency, output quality, and token consumption between the two models in the dashboard, and decide whether to officially switch after the evaluation is complete.
Scenario 4: Cost Accounting for Multi-Client Projects
A freelancer is working on projects for three clients simultaneously. Create a virtual key for each client project. At the end of the month, export the call volume and cost data for each key from the dashboard, directly serving as the basis for client billing.
Summary
The open-sourcing of NVIDIA NeMo Switchyard pushes intelligent model routing from concept to tangible engineering practice. The Stage Router's complexity-based tiered routing and protocol translation capabilities provide a valuable technical reference for the entire AI gateway field.
On the dimension of more complete engineering coverage, ServBay AI Gateway provides a corresponding answer. Channel management covers nearly 20 vendors and various relay stations; traffic scheduling supports priority, automatic failover, and hot switching; virtual keys achieve key security and multi-project isolation; model mapping makes underlying model switching transparent to the application layer; protocol conversion masks the format differences between OpenAI, Anthropic, and Gemini; usage statistics provide multi-dimensional cost tracking by model, by channel, and by project.
For developers and teams facing the challenge of managing multi-vendor AI APIs, these two projects offer solutions from different angles and are both worth paying attention to.