Inside New API: A Production AI Gateway's Inverted Index, Star-Topology Converter, and Redis-Backed Ledger
Teams operating self-hosted gateways across multiple LLM providers get a working blueprint for three problems that usually become bespoke glue: index-backed channel selection at request rates, protocol conversion that logs what it drops rather than silently losing fields, and a billing ledger that stays correct under concurrency. The source also demonstrates a clean consistency split, config converges eventually in 60 seconds, balances stay strongly consistent through Redis Lua, that most gateways hand-wave.
New API, the AGPLv3 fork of one-api, runs 47 channel adapters covering 40+ vendors, yet its production design reduces to three solved problems: who serves a request, how protocols speak to each other, and how money gets accounted. A single `/v1/chat/completions` request passes through seven stages, from multi-protocol auth to a retry loop whose failure decisions are logged as replayable policy events.
Two structural choices carry most of the engineering. Channel scheduling materializes an abilities table, an inverted index keyed by (group, model, channel_id) that turns "who can serve gpt-4o" into one indexed lookup, with retry count doubling as a priority-tier index so the Nth retry automatically drops to the Nth tier. Protocol conversion is a star topology centered on OpenAI Chat: 12 registered converters collapse to 6 real transformation pairs, with two-hop paths like Claude↔Gemini explicitly marked "discouraged" because every hop loses fields.
Billing is a true ledger lifecycle. The gateway pre-deducts an estimated deposit, settles against real upstream usage, and charges or refunds the delta through a two-phase commit that makes double-refunds impossible. Redis holds the authoritative balance through atomic Lua scripts while the database catches up 5 seconds later. Trusted users above a $10 threshold skip pre-deduction, settlement never fails (negative balances are accepted by design, capped by a single request's maximum bill), and a 5-second window exists where a stale DB read would be wrong if the fences didn't protect it.
Treating Redis as the authoritative ledger rather than a cache is the source's boldest call: for 5 seconds the DB is knowingly stale, and the code spends real effort (fences, version floors, TTL-only refreshes) making sure nothing reads that staleness into a wrong deduction.
The retry-count-as-priority-tier trick couples two existing quantities instead of adding a state object, which is elegant, but it quietly makes the retry policy and the routing policy inseparable.
Settlement's no-failure branch is a deliberate cost-behavior trade: the ceiling on a one-request bill makes unpaid orders survivable, and blocking every request on a pre-check would cost more in latency than the occasional negative balance costs in revenue.
The star topology for protocol conversion is the more honest answer to a many-format world: fewer converters and explicit "discouraged" labels on lossy paths beat trying to write perfect pairwise converters.