跪拜 Guibai
← Back to the summary

The AI Gateway That Turns Provider Switches into a Config Change

Series: "AI Application Production Handbook" Part 17 of 30 | Companion open-source project: https://github.com/ChenYingbo/ai-prod-demo This is Module 4 (Cost and Gateway) Part 4: C01 counts cost, C02 saves money, C03 routes traffic, C04 moves these three things out of business code.

1. The Problem: Optimizations Are Done, but the Pain Remains

C01–C03 work on the application side, but you'll hit three walls:

  1. Direct app-to-provider connection means changing providers requires rewriting code — switching from DeepSeek to OpenAI means touching every call site, and it's easy to miss some.
  2. Keys scattered everywhere — each service holds its own provider key. If one leaks, you don't know which service leaked it, and you can't revoke just that one key.
  3. Fallback, rate limiting, and cost tracking are rewritten in every application — reinventing the wheel, and each implementation differs, so reconciliation doesn't line up.

Solution: an AI gateway. Add a layer between the app and the provider, unify model names, centralize key management, standardize fallback and rate limiting, and persist cost data.

Core insight: the gateway is the first provider-agnostic layer, and the central point for key security and cost tracking. The application only knows logical model names (chat-strong / chat-cheap); however the provider changes, business code stays untouched.

2. Principles

2.1 Where the Gateway Sits

Application (only knows logical names chat-strong / chat-cheap)
        |
        v
   AI Gateway (LiteLLM)
   |-- model routing: logical name -> real provider model
   |-- key pool: virtual key management / per-key rate limiting / individual revocation
   |-- fallback: automatically switch to backup when primary fails
   |-- rate limiting: rpm/tpm/concurrency caps
   |-- cost tracking: spend persisted
        |
        v
   Providers (DeepSeek / OpenAI / ...)

2.2 Direct In-App vs. Gateway

Direct In-App Gateway
Changing providers Change code Change config (basis for C05 portability)
Key management Each app has its own key Centralized virtual keys, individually revocable
Fallback / rate limiting Written in each app Unified at gateway
Cost tracking Each app logs its own way Gateway persist spend logs

2.3 Five Capabilities (from this demo's actual config)

Capability Config Item Verification
Unified model names model_list (chat-strong → deepseek/deepseek-chat) [2/4] /v1/models
Virtual keys general_settings.master_key + PostgreSQL [3/4] /key/generate + /key/delete
Fallback litellm_params.fallbacks: ["chat-cheap"] Config layer (real trigger requires API key)
Rate limiting router_settings.max_parallel_requests: 10 Config layer
Cost tracking general_settings.database_url (PostgreSQL) [4/4] /spend/logs

2.4 Choosing a Gateway

Gateway Positioning Strengths Weaknesses
LiteLLM (this post) Open-source, self-hosted proxy OpenAI-compatible, 100+ providers, fallback/keys/cost built in Weak UI (API-centric)
Portkey Managed gateway Friendly UI, A/B testing built in Managed cost
Helicone Observability-focused Strong cost/latency analysis Weak routing/fallback

Selection logic: start with a self-hosted open-source option (LiteLLM) to learn the mechanics; when the team grows, evaluate managed options. The gateway isn't the most powerful, it's "sufficient + controllable."

2.5 A Hard Constraint: Must Use PostgreSQL

In the first version I set database_url to SQLite for convenience, and the gateway wouldn't start:

LiteLLM Proxy: DATABASE_URL uses unsupported scheme 'sqlite'.
LiteLLM's database features (virtual keys, spend tracking) require PostgreSQL.

Virtual keys and cost tracking only work with PostgreSQL. The fix is to reuse the existing postgres container in the compose setup and create a dedicated database (don't mix it with Langfuse):

docker compose exec postgres psql -U postgres -c "CREATE DATABASE litellm;"

3. Hands-On

3.1 Start the Gateway

cd demo
docker compose exec postgres psql -U postgres -c "CREATE DATABASE litellm;"   # first time
docker compose up -d litellm

3.2 Four Verification Checks (from real container logs)

python scripts/gateway_check.py

Real container verification excerpt (2026-11):

[1/4] health: 200 "I'm alive!" → OK
[2/4] models (3): chat-cheap, chat-strong, chat-strong-alt
[3/4] key/generate: OK (virtual key generated, then key/delete cleaned up)
[4/4] spend/logs: OK (0 records; will accumulate once real calls happen)
OK ✓ LiteLLM Gateway Ready

The four checks correspond to four things; missing one means the gateway isn't fully working:

  1. Health — /health/liveliness returns plain text "I'm alive!", not JSON (this tripped me up the first time)
  2. Model loading — /v1/models must show all three logical names; missing one means the YAML didn't load correctly
  3. Virtual keys — create and instantly delete to prove the key management plane works. Note: the generated key is actually valid, so delete it after verification
  4. Cost tracking — /spend/logs returns (0 records also counts as working). Once real calls happen, this grows; only then can you reconcile with the C01 cost model

3.3 Virtual Key Drill (the gateway's most valuable capability)

1. /key/generate creates a key specific to the "test environment"
2. Use it to call the gateway (permissions/budget apply only to this key)
3. If test environment leaks -> /key/delete revokes just that key
4. Production keys are completely unaffected

Compare: without a gateway, a leaked provider key exposes all services and you must rotate all keys. With a gateway, each environment has its own key, individually revocable and individually limited.

3.4 Line-by-Line Config (the most important lines)

model_list:
  - model_name: chat-strong                 # app only knows this name
    litellm_params:
      model: deepseek/deepseek-chat         # changing providers only changes this line
      api_key: os.environ/DEEPSEEK_API_KEY  # key not in config file
      fallbacks: ["chat-cheap"]             # gateway-level fallback (app-level is R04's degradation chain)

litellm_settings:
  num_retries: 2                            # automatic retry on transient failures (A01's gateway-side implementation)
  request_timeout: 60

router_settings:
  max_parallel_requests: 10                 # global concurrency cap, prevents overwhelming upstream

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: "postgresql://.../litellm"  # without it, virtual keys and cost tracking don't work

Note where fallbacks lives: it's inside litellm_params, not litellm_settings — both locations appear in official docs, but the wrong location doesn't error, just silently doesn't work, which is the hardest kind of bug to find.

4. Real Pitfalls

  1. Using SQLite as database_url — virtual keys / cost tracking only support PostgreSQL, fails at startup (tested with error)
  2. Master key leak — it's the parent permission for all virtual keys, effectively a master pass; in production it must be strong random and not committed to the codebase
  3. Fallback misconfigured at the wrong layer — writing it in litellm_settings doesn't error and doesn't take effect; only the verification script catches it
  4. No rate limiting — without max_parallel_requests, a traffic spike can directly overwhelm upstream; the gateway should be the place to stop that
  5. Cost tracking spinning — if database_url isn't set, /spend/logs returns 0 records forever, making you think there were no calls
  6. Taking the health endpoint for granted — /health/liveliness is plain text, /spend/logs in some versions returns a list directly; verification scripts must handle both

5. What You Learn This Part

Next (C05): End of Module 4, "Multi-Model Architecture and Vendor Lock-in" — "what if the provider raises prices or shuts down tomorrow?" How the architecture holds up (including a real provider switch walkthrough). Save and follow; one post a day, 30 days to get AI applications to production.

Comments

Top 1 from juejin.cn, machine-translated. The original thread is authoritative.

用户573468752399

I fully agree with the direction of this gateway layer. Let me add two details that only surface after you've scaled up: First, for fallback, route by error type. For 429, only back off and queue; for 401/402 and content moderation, don't retry at all, otherwise the primary-backup multiplication can increase your call count fivefold. Second, after switching, you need to trim the context. The system prompt's primary and backup should share the same prefix, otherwise cache hit rate drops from 80% to single digits, and the money saved on unit price won't make up for cache loss. For cost tracking, it's best to persist per call chain, including failed attempts; otherwise the cost of fallback will always be a black box. Several domestic models allow one key to call all, have enterprise channel pricing, and can export details by call chain. In production, that's how we've managed to keep fallback costs down. PM me for details.