Exact Caching: The Same Question Asked 100 Times Now Costs One Answer
For any service that serves many users asking similar questions, exact caching is the first cost optimization that doesn't degrade output. A 3ms cache hit versus a 1–11 second model call changes the capacity planning and cost model of a hot endpoint.
Repeated questions from different users or the same user repeatedly hit the model, multiplying cost and latency. Exact caching returns the previous answer for identical inputs, so the model is called once and every subsequent identical request reads from a store. The key includes the model name, the full system prompt, and the user prompt, so any change in those ingredients naturally invalidates the cache. This also isolates canary and stable versions because different prompts produce different keys.
The article walks through a 75-line SQLite implementation with a TTL invalidation strategy, unit tests for six cases, and a real end-to-end test. The first request, with a broken gateway, took 11,151 ms through the fallback chain; the second identical request hit the cache and returned in 3 ms. A hit rate formula ties directly to the average cost per request from the previous module, turning 'how much did we save' from a guess into a number.
The pitfalls are concrete: forgetting model or system in the key, caching across gray releases, no TTL after knowledge base updates, not logging hits and misses, storing sensitive answers in plaintext, and cache storms when a hot key expires all at once. The recommendation is to start with exact caching before semantic or prefix caching, and to prefer over-invalidation over stale answers.
Including the full system prompt in the key gives gray release isolation for free, so a single cache design handles both cost and release safety.
The order of implementing cache types matters: starting with exact cache avoids false-hit complexity, and reversing the order creates avoidable incidents.
The cost formula ties directly to per-request average cost, making caching ROI measurable before implementation rather than a post-hoc guess.
The 11s-to-3ms contrast is exaggerated by the fallback chain, but the constant-time hit path is a real capacity win for high-QPS endpoints.