The AI Gateway That Turns Provider Switches into a Config Change
Series: "AI Application Production Handbook" Part 17 of 30 | Companion open-source project: https://github.com/ChenYingbo/ai-prod-demo This is Module 4 (Cost and Gateway) Part 4: C01 counts cost, C02 saves money, C03 routes traffic, C04 moves these three things out of business code.
1. The Problem: Optimizations Are Done, but the Pain Remains
C01–C03 work on the application side, but you'll hit three walls:
- Direct app-to-provider connection means changing providers requires rewriting code — switching from DeepSeek to OpenAI means touching every call site, and it's easy to miss some.
- Keys scattered everywhere — each service holds its own provider key. If one leaks, you don't know which service leaked it, and you can't revoke just that one key.
- Fallback, rate limiting, and cost tracking are rewritten in every application — reinventing the wheel, and each implementation differs, so reconciliation doesn't line up.
Solution: an AI gateway. Add a layer between the app and the provider, unify model names, centralize key management, standardize fallback and rate limiting, and persist cost data.
Core insight: the gateway is the first provider-agnostic layer, and the central point for key security and cost tracking. The application only knows logical model names (chat-strong / chat-cheap); however the provider changes, business code stays untouched.
2. Principles
2.1 Where the Gateway Sits
Application (only knows logical names chat-strong / chat-cheap)
|
v
AI Gateway (LiteLLM)
|-- model routing: logical name -> real provider model
|-- key pool: virtual key management / per-key rate limiting / individual revocation
|-- fallback: automatically switch to backup when primary fails
|-- rate limiting: rpm/tpm/concurrency caps
|-- cost tracking: spend persisted
|
v
Providers (DeepSeek / OpenAI / ...)
2.2 Direct In-App vs. Gateway
| Direct In-App | Gateway | |
|---|---|---|
| Changing providers | Change code | Change config (basis for C05 portability) |
| Key management | Each app has its own key | Centralized virtual keys, individually revocable |
| Fallback / rate limiting | Written in each app | Unified at gateway |
| Cost tracking | Each app logs its own way | Gateway persist spend logs |
2.3 Five Capabilities (from this demo's actual config)
| Capability | Config Item | Verification |
|---|---|---|
| Unified model names | model_list (chat-strong → deepseek/deepseek-chat) |
[2/4] /v1/models |
| Virtual keys | general_settings.master_key + PostgreSQL |
[3/4] /key/generate + /key/delete |
| Fallback | litellm_params.fallbacks: ["chat-cheap"] |
Config layer (real trigger requires API key) |
| Rate limiting | router_settings.max_parallel_requests: 10 |
Config layer |
| Cost tracking | general_settings.database_url (PostgreSQL) |
[4/4] /spend/logs |
2.4 Choosing a Gateway
| Gateway | Positioning | Strengths | Weaknesses |
|---|---|---|---|
| LiteLLM (this post) | Open-source, self-hosted proxy | OpenAI-compatible, 100+ providers, fallback/keys/cost built in | Weak UI (API-centric) |
| Portkey | Managed gateway | Friendly UI, A/B testing built in | Managed cost |
| Helicone | Observability-focused | Strong cost/latency analysis | Weak routing/fallback |
Selection logic: start with a self-hosted open-source option (LiteLLM) to learn the mechanics; when the team grows, evaluate managed options. The gateway isn't the most powerful, it's "sufficient + controllable."
2.5 A Hard Constraint: Must Use PostgreSQL
In the first version I set database_url to SQLite for convenience, and the gateway wouldn't start:
LiteLLM Proxy: DATABASE_URL uses unsupported scheme 'sqlite'.
LiteLLM's database features (virtual keys, spend tracking) require PostgreSQL.
Virtual keys and cost tracking only work with PostgreSQL. The fix is to reuse the existing postgres container in the compose setup and create a dedicated database (don't mix it with Langfuse):
docker compose exec postgres psql -U postgres -c "CREATE DATABASE litellm;"
3. Hands-On
3.1 Start the Gateway
cd demo
docker compose exec postgres psql -U postgres -c "CREATE DATABASE litellm;" # first time
docker compose up -d litellm
3.2 Four Verification Checks (from real container logs)
python scripts/gateway_check.py
Real container verification excerpt (2026-11):
[1/4] health: 200 "I'm alive!" → OK
[2/4] models (3): chat-cheap, chat-strong, chat-strong-alt
[3/4] key/generate: OK (virtual key generated, then key/delete cleaned up)
[4/4] spend/logs: OK (0 records; will accumulate once real calls happen)
OK ✓ LiteLLM Gateway Ready
The four checks correspond to four things; missing one means the gateway isn't fully working:
- Health —
/health/livelinessreturns plain text"I'm alive!", not JSON (this tripped me up the first time) - Model loading —
/v1/modelsmust show all three logical names; missing one means the YAML didn't load correctly - Virtual keys — create and instantly delete to prove the key management plane works. Note: the generated key is actually valid, so delete it after verification
- Cost tracking —
/spend/logsreturns (0 records also counts as working). Once real calls happen, this grows; only then can you reconcile with the C01 cost model
3.3 Virtual Key Drill (the gateway's most valuable capability)
1. /key/generate creates a key specific to the "test environment"
2. Use it to call the gateway (permissions/budget apply only to this key)
3. If test environment leaks -> /key/delete revokes just that key
4. Production keys are completely unaffected
Compare: without a gateway, a leaked provider key exposes all services and you must rotate all keys. With a gateway, each environment has its own key, individually revocable and individually limited.
3.4 Line-by-Line Config (the most important lines)
model_list:
- model_name: chat-strong # app only knows this name
litellm_params:
model: deepseek/deepseek-chat # changing providers only changes this line
api_key: os.environ/DEEPSEEK_API_KEY # key not in config file
fallbacks: ["chat-cheap"] # gateway-level fallback (app-level is R04's degradation chain)
litellm_settings:
num_retries: 2 # automatic retry on transient failures (A01's gateway-side implementation)
request_timeout: 60
router_settings:
max_parallel_requests: 10 # global concurrency cap, prevents overwhelming upstream
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
database_url: "postgresql://.../litellm" # without it, virtual keys and cost tracking don't work
Note where fallbacks lives: it's inside litellm_params, not litellm_settings — both locations appear in official docs, but the wrong location doesn't error, just silently doesn't work, which is the hardest kind of bug to find.
4. Real Pitfalls
- Using SQLite as
database_url— virtual keys / cost tracking only support PostgreSQL, fails at startup (tested with error) - Master key leak — it's the parent permission for all virtual keys, effectively a master pass; in production it must be strong random and not committed to the codebase
- Fallback misconfigured at the wrong layer — writing it in
litellm_settingsdoesn't error and doesn't take effect; only the verification script catches it - No rate limiting — without
max_parallel_requests, a traffic spike can directly overwhelm upstream; the gateway should be the place to stop that - Cost tracking spinning — if
database_urlisn't set,/spend/logsreturns 0 records forever, making you think there were no calls - Taking the health endpoint for granted —
/health/livelinessis plain text,/spend/logsin some versions returns a list directly; verification scripts must handle both
5. What You Learn This Part
- Can draw the gateway's position in production architecture and name the five capabilities (routing/keys/fallback/rate limiting/cost)
- Can compare trade-offs between "direct in-app" and "gateway"
- Can compare LiteLLM / Portkey / Helicone for selection
-
gateway_check.pypasses 4/4 (real container verification) - Completed the virtual key drill: create → use → individual revocation
- Remember the hard constraint: SQLite is not supported
Next (C05): End of Module 4, "Multi-Model Architecture and Vendor Lock-in" — "what if the provider raises prices or shuts down tomorrow?" How the architecture holds up (including a real provider switch walkthrough). Save and follow; one post a day, 30 days to get AI applications to production.
Top 1 from juejin.cn, machine-translated. The original thread is authoritative.
I fully agree with the direction of this gateway layer. Let me add two details that only surface after you've scaled up: First, for fallback, route by error type. For 429, only back off and queue; for 401/402 and content moderation, don't retry at all, otherwise the primary-backup multiplication can increase your call count fivefold. Second, after switching, you need to trim the context. The system prompt's primary and backup should share the same prefix, otherwise cache hit rate drops from 80% to single digits, and the money saved on unit price won't make up for cache loss. For cost tracking, it's best to persist per call chain, including failed attempts; otherwise the cost of fallback will always be a black box. Several domestic models allow one key to call all, have enterprise channel pricing, and can export details by call chain. In production, that's how we've managed to keep fallback costs down. PM me for details.