AI Gateway: Turning 'Switch to DeepSeek' into a Config Change
Foreword
Hi, I'm Su San.
"A while back, a member of my community told me about an AI customer service system they built. Just for the 'calling the model' step, they wrote three adapters in the business layer: one for OpenAI, one for Anthropic, and one for Qwen (Tongyi Qianwen). Each adapter had its own retry logic, its own error code mapping, and its own token statistics.
I asked him: 'If your boss says tomorrow that DeepSeek is cheaper and you should switch everything to it, how many places do you need to change?'
He smiled wryly: 'All three adapters would need to change, and there are a bunch of if provider == 'openai' checks scattered in the business code.'
This is exactly the problem AI Gateway solves.
It's not 'just another middleware' – it pulls 'calling the model' out of business code and turns it into a unified infrastructure layer.
You talk to one address; routing, translation, fallback, and billing are left to the gateway.
Today's article walks through what AI Gateway really solves, its core principles, and how to use it, from start to finish.
I hope it helps.
More hands-on projects on Java Assault Team website: susan.net.cn/project
1. Why 'calling the model directly' turns into a disaster?
Some people might say: 'Can't I just call the OpenAI API directly and add a try-catch?'
When you have a single model and a single provider, that works fine. But once your system starts to 'grow', the problems start to surface.
Problem one: protocols are inconsistent.
OpenAI uses the tool_calls field, Anthropic uses tool_use content blocks, Gemini uses functionCall parts.
Every time you integrate a new provider, you write a set of input conversion, output conversion, and streaming event conversions. Three adapters add up to hundreds of lines of glue code.
Problem two: scattered failure handling.
If a provider suddenly returns 5xx, you want to fall back to a backup model automatically.
But if you write try-catch plus fallback in every business method, the code becomes extremely ugly. Moreover, 'when to retry, when to degrade, how to handle 429 rate limits' – the logic may differ from place to place.
Problem three: costs aren't visible.
Your app calls both OpenAI and DeepSeek. When the monthly bill arrives, you want to know 'how much money each business line spent', but the billing formats from the two providers don't match, and you can't reconcile them.
Problem four: scattered security management.
You want to set different token limits per department, redact PII from user input, and audit every model call.
If every application manages these individually, you can't do them uniformly.
The essence of AI Gateway is to pull these problems out of business code and solve them in one unified place.
2. One diagram shows where AI Gateway sits
The business application only talks to the Gateway in one 'language' (usually the OpenAI Chat Completions format). The Gateway translates it into something each provider understands, then, based on your policy, decides where to send the request.
The core value of this design: business code is decoupled from providers. Today you use OpenAI, tomorrow you switch to DeepSeek, the day after you add a self-hosted Ollama, and your business code doesn't change a single line. You only need to change the Gateway's configuration.
3. What does AI Gateway actually do?
3.1 Protocol normalization: get all providers to 'speak Mandarin'
This is the most basic and most tedious part of an AI Gateway.
The differences between provider APIs are deeper than they look.
Take tool calling: OpenAI uses tool_calls, Anthropic uses tool_use content blocks, Gemini uses functionCall parts.
The placement of the system prompt is also different.
The incremental structure of streaming responses is different as well.
The Gateway acts as the translator.
You send a standard OpenAI-format request in; the Gateway, based on routing rules, translates it to the target provider's format and sends it out; when it receives the response, it translates it back to the standard format and returns it to you.
This means your code only needs to recognize one protocol.
When you change providers, you don't need to rewrite any request construction logic.
LiteLLM's documentation says it clearly: litellm.completion() is responsible for 'Translation and parameter mapping'.
3.2 Model routing: choose 'who answers' based on policy
This is the most technically interesting part of an AI Gateway. Routing strategies range from simple to complex:
- Fixed model: hard-coded, changed manually. Simplest and least flexible.
- Rule-based routing: map models by scenario, tenant, risk level. For example, 'use a cheap model for internal testing, a flagship model for production.'
- Cost-based routing: try a small model first, escalate to a stronger model if it's insufficient. For example, use DeepSeek first; if it returns low confidence, hand off to GPT-4o for a re-answer.
- Fallback and circuit breaking: not all errors should trigger fallback. Network blips and 5xx server errors are suitable for switching; 429 rate limits require caution – you need to read
Retry-After; auth errors are useless to switch on.
LiteLLM's Router handles load balancing, fallbacks, and retries at this layer.
3.3 Key management: mapping virtual keys to real keys
This is a hard requirement for enterprise scenarios.
Your application shouldn't directly hold an OpenAI key.
The right approach: the Gateway issues a 'virtual key' to each application or user. The application calls the Gateway with the virtual key, and the Gateway looks up the corresponding real provider key and forwards the request.
In LiteLLM's request flow, the first step checks whether the Bearer token's corresponding virtual key is valid and within budget. You can revoke a virtual key at any time without affecting others; you can track 'which app called how many times and spent how much'; you can set different budget caps per department.
3.4 Observability: token and cost are the new QPS
Traditional API gateways watch QPS, latency, and error rates.
AI Gateway watches token consumption, cost attribution, and time to first token.
These metrics have completely different meanings.
Two requests may have the same QPS, but one request consumes 100 tokens and another consumes 100,000 tokens, so the cost difference is a thousandfold. Traditional QPS-based rate limiting is nearly useless here.
AI Gateway rate-limits by tokens and controls by cost budgets; that's the correct governance for AI scenarios.
Alibaba Cloud's AI Gateway documentation lists token metering and billing and semantic caching as two parallel core capabilities.
4. Run an AI Gateway with LiteLLM
Some people may say: 'You've talked a lot, but how do I actually use it?'
LiteLLM is currently the most widely used open-source AI Gateway.
Thoughtworks' Technology Radar moved it from 'languages and frameworks' to 'platforms', saying "we see it more as infrastructure than a library."
4.1 Installation and startup
pip install 'litellm[proxy]'
litellm --config /path/to/config.yaml
4.2 Config file: define model routing
model_list:
- model_name: "gpt-4o"
litellm_params:
model: "openai/gpt-4o"
api_key: os.environ/OPENAI_API_KEY
- model_name: "deepseek-chat"
litellm_params:
model: "deepseek/deepseek-chat"
api_key: os.environ/DEEPSEEK_API_KEY
- model_name: "qwen-max"
litellm_params:
model: "dashscope/qwen-max"
api_key: os.environ/DASHSCOPE_API_KEY
4.3 Client call: no code changes at all
from openai import OpenAI
client = OpenAI(
api_key="sk-litellm-virtual-key",
base_url="http://localhost:4000"
)
response = client.chat.completions.create(
model="gpt-4o", # or "deepseek-chat" or "qwen-max"
messages=[{"role": "user", "content": "Explain what an AI Gateway is"}]
)
Your business code only knows base_url and model. Whether it's OpenAI or DeepSeek underneath, the business code has no idea.
5. Use AI to help troubleshoot gateway config problems
An AI Gateway is itself a 'middle layer'; when problems occur, the troubleshooting chain is longer than a direct connection. But because it's a gateway, its logs are centralized and structured, so AI can help you locate issues quickly.
Suppose your application suddenly reports that a model is unavailable. How do you quickly determine whether it's a gateway config problem or a provider problem?
Traditional approach: look at application logs, gateway logs, and the provider status page; compare all three – ten minutes have passed.
AI-assisted approach: feed the gateway request logs and error logs to an AI and say:
"This is my LiteLLM gateway log. When the app calls
gpt-4o, it returns 500, but callingdeepseek-chatworks fine. Help me analyze whether it's a gateway config problem or an OpenAI-side problem."
The AI scans the logs, finds that the gpt-4o request failed at the gateway side, with AuthenticationError: Incorrect API key.
It tells you: "The gateway's key configuration to OpenAI has a problem, not that the OpenAI service is down. Check whether the OPENAI_API_KEY environment variable is loaded correctly."
What AI can't do for you: determine why the key is invalid – is it expired, revoked, or was the environment variable not injected? That requires you to look at the provider console and deployment config.
AI narrows the scope; you make the final judgment.
6. Pros and cons
Pros
1. Decoupling business from providers
Business code only recognizes one protocol and one address. Switch providers or add models by changing gateway config; business code stays put.
2. Centralized failure handling
Fallback, retry, circuit breaking, rate limiting – all configured at the gateway layer. No need for each application to write its own.
3. Observable and controllable costs
Token consumption, cost attribution, budget caps – managed at the gateway layer. You know where the money goes and to whom.
4. Safer key management
Virtual keys map to real keys; applications don't hold provider secrets directly. Rotation, revocation, and permission control all happen at the gateway layer.
5. Automated protocol translation
The format differences between OpenAI, Anthropic, and Gemini are translated by the gateway. You don't write three adapters.
Cons
1. An extra layer means an extra point of failure
If the gateway itself goes down, no application can call models. So the gateway's high availability is mandatory; it can't be a single point.
2. Provider-specific features may be 'flattened'
LiteLLM's drop_params mode silently drops unsupported parameters. If you rely on a provider-specific capability, you may need to bypass the gateway for that direct connection, or accept 'feature degradation when crossing providers.'
3. Self-hosting carries operational costs
Open-source gateways (like LiteLLM) need to be deployed, kept available, and patched for security. Thoughtworks notes that LiteLLM had already reported over 25 CVEs in 2026, requiring timely patching.
4. Not every scenario needs it
A system that connects to one model, has one caller, and doesn't care about cost details – adding a gateway is overengineering.
7. Applicable scenarios
| Scenario | Recommendation | Reason |
|---|---|---|
| Multi-model / multi-provider switching | ✅✅✅ Strongly recommended | Protocol translation + routing, zero business code changes |
| Cost control and attribution | ✅✅✅ Strongly recommended | Token-level rate limiting, budget control, cost attribution |
| Enterprise key management and auditing | ✅✅✅ Strongly recommended | Virtual keys, audit logs, permission control |
| Automatic fallback when a provider fails | ✅✅✅ Strongly recommended | Fallback, circuit breaking, retries centrally configured |
| Single-model simple app | ❌ Not recommended | Gateway is overengineering |
| Relies on provider-specific features | ⚠️ Needs evaluation | Gateway may flatten differences; confirm support scope |
More hands-on projects on Java Assault Team website: susan.net.cn/project
8. Final words
Back to the original question: Why are more and more people using AI Gateway?
Because 'calling the model' is shifting from 'writing a line of code' to 'managing a piece of infrastructure.'
When your system connects to only one model, calling the API directly is fine.
But when your system involves three providers, five models, a dozen applications, and dozens of users, 'how to call' becomes a matter that needs unified governance.
Inconsistent protocols, scattered failure handling, invisible costs, and unmanageable keys – each of these on its own is easy to solve, but together they add up to thousands of lines of glue code and countless late-night emergency calls.
The value of AI Gateway is to pull these 'cross-cutting concerns' out of business code and solve them in one unified place. Business code only cares about 'what I want'; the gateway handles 'how to get it.'
Of course, it's not a silver bullet.
An extra layer means an extra point of failure; self-hosting brings operational costs; provider-specific features may be flattened.
But for teams that run multiple models, multiple providers, and need cost control and auditing, it has already shifted from an 'option' to 'infrastructure.'
Related recommendations: