# NVIDIA Switchyard: A Rust-Based Proxy for Unified LLM Traffic Routing and API Translation
## The Multi-Provider Routing Problem in Modern AI Workflows
Modern development teams face a persistent challenge when building coding agents. Popular agent frameworks like Claude Code communicate through the Anthropic Messages API, while others such as OpenAI’s Codex CLI rely on OpenAI-compatible endpoints. Meanwhile, the large language models organizations actually want to host often run behind specialized serving stacks like vLLM, NVIDIA NIM, or Ollama, each with its own API conventions. Rewriting the agent code to accommodate every backend is neither practical nor maintainable, which means the burden of translation has to be handled at an infrastructure layer somewhere in the stack.
NVIDIA’s response to this problem is Switchyard, a proxy and library built in Rust designed to sit between client applications and upstream model providers. It handles request routing across multiple backends, translates between different API formats transparently, records detailed operational metrics, and exposes composable routing algorithms that teams can customize to their specific needs. The project is open-source under the Apache 2.0 license, with documentation available through NVIDIA’s developer portal.
## How the Architecture Works
At its core, Switchyard acts as a universal intermediary. When a client sends a request using its native API format — whether that is OpenAI Chat Completions, OpenAI Responses, or Anthropic Messages — Switchyard decodes the incoming payload into provider-neutral Rust types internally. A routing algorithm then selects an appropriate backend, and the request is re-encoded into that backend’s specific wire format before being forwarded. The response travels back through the same pipeline, with Switchyard translating it (including streaming events) back into the format the original client expects.
This design introduces a powerful decoupling: the API format an agent was written against no longer needs to match the format spoken by the model server it is ultimately calling. A single Switchyard instance can accept requests in any of the three supported inbound formats and route each one to a backend using a completely different format. An Anthropic Messages client can talk to an OpenAI-compatible model server, and vice versa, without any changes to either the client application or the upstream model.
## Three Deployment Modes
Switchyard offers three distinct ways to integrate it into a workflow, each suited to a different use case.
The **launcher path** is designed with coding agents in mind. Teams install the published command-line tool using Python’s package manager and then invoke it with a simple command — `switchyard launch claude`, `switchyard launch codex`, or `switchyard launch openclaw` — pointing it at either a prepackaged deployment configuration or a custom TOML file they define themselves.
The **server path** installs a standalone proxy binary directly from Rust’s package registry. Once installed, operators can validate their configuration using a dry-run mode before starting the server on a host and port of their choosing. This approach is ideal for teams that want a centralized proxy service accessible by multiple clients simultaneously.
The **library path** embeds Switchyard’s routing logic directly into a Rust application through `switchyard-libsy`. In this mode, the library does not spin up its own HTTP server and never communicates with a model on its own behalf. Instead, it evaluates routing decisions and hands each model call back to the calling application, giving developers full control over how requests are dispatched.
## Routing Algorithms Under the Hood
A route in Switchyard consists of a single client-visible model identifier paired with the algorithm that governs how requests assigned to that route are handled. The system ships with four routing strategies, each addressing a different operational need.
The simplest option is **passthrough**, which forwards every request to a single designated target without any decision-making logic. For teams that need to split traffic across multiple backends for A/B testing or cost optimization, **random** routing distributes requests across targets using optional relative weights and an optional seed value that makes the selection sequence reproducible.
The **LLM classifier** route type introduces intelligence into the routing decision. It sends each request to a designated classifier target that evaluates the task and returns a capability verdict, after which the request is routed to either a weak target or a strong target based on that assessment. Several configuration parameters allow fine-tuning, including a required base threshold, a minimum confidence level, an elevated floor for heightened capabilities, and session affinity settings to maintain consistency within a conversation. When the classifier cannot confidently make a determination, the request defaults to the strong target. An escalation mode is also available, where every turn first goes to the weaker tier and a judge decides whether it should be rerun on the stronger tier.
The fourth strategy, **stage router**, takes a different approach by scoring tool-result and agent-progress signals from recent conversation turns to decide between a capable target and an efficient one. This avoids the overhead of an extra classifier call on most turns, making it well-suited for scenarios where latency matters and the routing decision can be inferred from the agent’s recent behavior.
An important design principle is that strong, weak, capable, and efficient are roles assigned within a route rather than inherent properties of a specific model. The same upstream model instance can serve different roles across different routes, giving teams maximum flexibility in how they allocate capacity.
## Observability and Metrics
Switchyard integrates with OpenTelemetry to expose Prometheus-formatted metrics at the `/metrics` endpoint. The metric families cover a comprehensive set of operational data, including total request counts, error counts, model-call latency, full-turn latency, and detailed token breakdowns for prompt tokens, completion tokens, cached tokens, cache-creation tokens, and reasoning tokens. Metrics also track upstream HTTP attempts segmented by outcome and response code.
One of the most analytically useful metrics is `switchyard_routing_overhead_ms`, which measures the time the routing algorithm itself consumes, isolated from the model-call latency. For classifier-based routes, this metric captures the classification time since classifier calls are not deducted from model-call measurement. For passthrough and random routes, it reports the sub-millisecond cost of target selection. A `tier` label distinguishes strong and weak decisions made by classifier routes, while a separate JSON-based routing log file and a per-session statistics endpoint provide additional visibility into call volumes and token usage at the session level.
## Configuration Model
Deployment is driven by a TOML configuration file organized into three conceptual layers. The `llm_clients` section defines base URLs, wire formats, credential environment variables, and retry policies. The `targets` section binds specific upstream model identifiers to their corresponding client definitions. The `routes` section exposes client-visible model IDs and associates each one with its routing algorithm. Secrets are never stored directly in the configuration file — credentials are referenced through environment variable names in the `api_key_env` field, and retry logic defaults to two attempts covering transport failures, timeouts, HTTP 408 and 429 responses, and server errors in the 5xx range.
## Current Status and Limitations
Switchyard is explicitly labeled as pre-alpha and experimental by NVIDIA. The binary is available for installation from crates.io and the launcher from PyPI, and it can self-host on any machine that meets its runtime requirements. However, the project is not intended for production workloads. NVIDIA warns that the API, routing algorithms, and overall behavior are expected to undergo significant changes before reaching version 1.0. Teams evaluating the tool should treat it strictly as an evaluation and experimentation platform at this stage.
## Key Takeaways
– Switchyard is an open-source Rust proxy and library that routes and translates LLM traffic between different API formats.
– It supports bidirectional translation between OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages, including streaming responses.
– Four routing strategies are available: passthrough, random, LLM-classifier, and signal-driven stage router.
– Detailed Prometheus metrics isolate routing overhead from model-call latency, segmented by model and tier.
– The project is in a pre-alpha, experimental state and should not be used in production environments.
—
## Frequently Asked Questions
**What problem does Switchyard solve?**
Switchyard solves the API format mismatch problem that arises when teams use agent frameworks that communicate in one protocol (such as Anthropic Messages or OpenAI Chat Completions) while hosting their models behind backends that speak a different protocol (such as vLLM, NVIDIA NIM, or Ollama). It eliminates the need to rewrite agent code to accommodate every backend.
**What formats does Switchyard support?**
It accepts three inbound formats — OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages — and can route each of those to any upstream backend format, translating in both directions including streaming events.
**Is Switchyard ready for production use?**
No. NVIDIA explicitly classifies Switchyard as pre-alpha and experimental. It is intended for evaluation and testing purposes only, and the API and routing algorithms are expected to change substantially before a stable release.
**How many routing algorithms does Switchyard include?**
Four: passthrough (single target), random (weighted distribution across targets), llm_classifier (uses a judge model to pick weak or strong targets), and stage_router (scores recent tool and progress signals to select between capable and efficient targets).
**Can the same model serve different roles in different routes?**
Yes. Strong, weak, capable, and efficient are roles defined within a route, not fixed attributes of a model. A single upstream model can be assigned different roles across different routes.
**How are secrets managed in the configuration?**
Credentials are never stored directly in the TOML file. Each client definition references an environment variable by name, and Switchyard reads the actual secret from that environment variable at runtime.
**What kind of metrics are available?**
Switchyard exposes Prometheus metrics covering request counts, error counts, token usage (prompt, completion, cached, cache-creation, and reasoning), model-call latency, full-turn latency, routing overhead, upstream HTTP attempts, and a tier label for classifier decisions. A JSON routing log and a per-session stats endpoint provide additional granularity.
**How can teams deploy Switchyard?**
There are three deployment paths: a launcher tool for coding agents (`switchyard launch`), a standalone proxy server installed via Cargo, and an embeddable Rust library (`switchyard-libsy`) for applications that want to own their HTTP stack.
**What license is Switchyard released under?**
Switchyard is released under the Apache 2.0 license.
—
## Conclusion
NVIDIA Switchyard addresses a real and growing pain point in the AI infrastructure landscape. As teams adopt multiple agent frameworks and host models across diverse serving backends, the need for a translation and routing layer becomes unavoidable. By building this layer as a Rust-based proxy with a clean separation of concerns — decoding, routing, encoding, and translating — Switchyard offers a flexible foundation that teams can evaluate and integrate into their workflows without modifying their existing agent code. The composable routing algorithms and detailed observability metrics make it a compelling tool for experimentation, especially for teams exploring cost optimization strategies, A/B testing across model tiers, or intelligent routing based on task complexity. While its pre-alpha status means it is not yet suitable for production, the architectural direction it points toward — provider-neutral routing with transparent format translation — addresses a genuine gap in the current ecosystem of LLM serving tools.
Thank you for reading
