跪拜 Guibai
← All articles
Artificial Intelligence · AI-Generated Content · Agent

The AI Gateway That Turns Provider Switches into a Config Change

By 知行砚 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

With a gateway in place, switching from DeepSeek to OpenAI is a configuration change, not a code audit, and a leaked developer key can be revoked per environment instead of forcing a global rotation. Cost tracking gets a single source of truth, making the expense model from earlier modules actually reconcilable.

Summary

Applications that talk directly to model providers pay three costs: switching from DeepSeek to OpenAI means touching every call site, provider keys are scattered so a leak can't be traced to one service, and fallback, rate limiting, and cost accounting get reimplemented differently in each app. An AI gateway solves all three by inserting a layer that maps logical model names to real providers, manages virtual keys per environment, applies fallback and concurrency caps centrally, and writes spend data to a database.

LiteLLM is the reference implementation in this guide. Its configuration points are explicit: model_list maps chat-strong to deepseek/deepseek-chat, master_key plus PostgreSQL power virtual keys, fallbacks live under litellm_params (not litellm_settings, where they silently fail), max_parallel_requests sets a global ceiling, and database_url must point to PostgreSQL or the proxy won't even start. The guide verifies the gateway with four checks: health, model listing, key generate/delete, and spend logs, each corresponding to one capability.

The real value shows up in the virtual key drill: generate a test-environment key, use it, then delete it individually while production keys stay untouched. Without a gateway, a leaked key forces a full rotation of all services. The article also warns about six pitfalls, from SQLite rejection to the health endpoint returning plain text instead of JSON, and confirms that the next article will walk through handling vendor lock-in.

Takeaways
— Applications reference logical model names like chat-strong and chat-cheap; the real provider model is mapped only in the gateway's YAML.
— The gateway's virtual keys are per-environment: generate one for test, use it, and delete it without touching production keys.
— LiteLLM's virtual keys and spend tracking require PostgreSQL; setting database_url to SQLite makes the proxy refuse to start.
— The fallbacks parameter belongs under litellm_params, not litellm_settings; the wrong location gives no error and no effect.
— The /health/liveliness endpoint returns plain text "I'm alive!", not JSON, so verification scripts must accept both.
— Without max_parallel_requests, a traffic spike can hammer the upstream provider; the gateway is the right place to enforce a global concurrency cap.
— The four-step verification (health, model listing, key generate/delete, spend logs) confirms the gateway is actually functional.
Conclusions

The gateway moves provider-switch risk from code to configuration, but that only helps if the logical model names are stable and well-documented.

The silent failure of a misplaced fallbacks setting is a strong argument for automated verification scripts in any gateway deployment.

Requiring PostgreSQL for operational state is a sign that AI infrastructure is maturing: you now need a real database for keys and spend data, not just a cache.

Choosing a self-hosted gateway like LiteLLM first teaches the mechanics; moving to a managed option only makes sense once the team knows what it needs.

Concepts & terms
AI gateway
A service layer between applications and AI model providers that centralizes model routing, key management, rate limiting, fallback, and cost tracking, so apps treat providers as interchangeable.
Virtual key
A generated API key that represents a set of permissions and limits (e.g., per environment) and can be individually revoked, instead of exposing the provider's master key.
Logical model name
An application-facing alias (such as chat-strong) that the gateway maps to a real provider model; apps never see the actual model ID.
Fallback
A configured secondary model that is automatically used when the primary model fails or times out, implemented at the gateway level.
From the discussion
Featured comments
用户573468752399

I fully agree with the direction of this gateway layer. Let me add two details that only surface after you've scaled up: First, for fallback, route by error type. For 429, only back off and queue; for 401/402 and content moderation, don't retry at all, otherwise the primary-backup multiplication can increase your call count fivefold. Second, after switching, you need to trim the context. The system prompt's primary and backup should share the same prefix, otherwise cache hit rate drops from 80% to single digits, and the money saved on unit price won't make up for cache loss. For cost tracking, it's best to persist per call chain, including failed attempts; otherwise the cost of fallback will always be a black box. Several domestic models allow one key to call all, have enterprise channel pricing, and can export details by call chain. In production, that's how we've managed to keep fallback costs down. PM me for details.

See top comments, translated →
Source: juejin.cn ↗ Google Translate ↗ Backup ↗