跪拜 Guibai
← All articles
Artificial Intelligence · AIGC · Agent

Exact Caching: The Same Question Asked 100 Times Now Costs One Answer

By 知行砚 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

For any service that serves many users asking similar questions, exact caching is the first cost optimization that doesn't degrade output. A 3ms cache hit versus a 1–11 second model call changes the capacity planning and cost model of a hot endpoint.

Summary

Repeated questions from different users or the same user repeatedly hit the model, multiplying cost and latency. Exact caching returns the previous answer for identical inputs, so the model is called once and every subsequent identical request reads from a store. The key includes the model name, the full system prompt, and the user prompt, so any change in those ingredients naturally invalidates the cache. This also isolates canary and stable versions because different prompts produce different keys.

The article walks through a 75-line SQLite implementation with a TTL invalidation strategy, unit tests for six cases, and a real end-to-end test. The first request, with a broken gateway, took 11,151 ms through the fallback chain; the second identical request hit the cache and returned in 3 ms. A hit rate formula ties directly to the average cost per request from the previous module, turning 'how much did we save' from a guess into a number.

The pitfalls are concrete: forgetting model or system in the key, caching across gray releases, no TTL after knowledge base updates, not logging hits and misses, storing sensitive answers in plaintext, and cache storms when a hot key expires all at once. The recommendation is to start with exact caching before semantic or prefix caching, and to prefer over-invalidation over stale answers.

Takeaways
— The cache key must include model, system prompt, and user prompt; omitting any causes stale answers or cross-version contamination.
— Hit rate = hits / (hits + misses); cost savings = hit count × average cost per request.
— TTL is the default invalidation; active invalidation is for knowledge bases; no invalidation is for static content.
— A 75-line SQLite implementation works; swapping to Redis requires no interface changes.
— In a real test, the first miss took 11,151 ms (due to fallback chain) and the hit took 3 ms.
— Six unit tests cover round-trip, key differentiation, TTL expiry, hit counting, no LLM call on hit, and log-based stats.
— Pitfalls include missing key components, gray release contamination, no TTL, no hit logging, plaintext sensitive answers, and cache storms.
Conclusions

Including the full system prompt in the key gives gray release isolation for free, so a single cache design handles both cost and release safety.

The order of implementing cache types matters: starting with exact cache avoids false-hit complexity, and reversing the order creates avoidable incidents.

The cost formula ties directly to per-request average cost, making caching ROI measurable before implementation rather than a post-hoc guess.

The 11s-to-3ms contrast is exaggerated by the fallback chain, but the constant-time hit path is a real capacity win for high-QPS endpoints.

Concepts & terms
Exact caching
A cache that returns a stored response only when the input is byte-for-byte identical, typically keyed by a hash of model, system prompt, and user prompt.
Semantic caching
A cache that returns a stored response for semantically similar inputs, using embeddings and vector search to match paraphrases or related questions.
Prefix caching
A caching mechanism that shares computation or stored results for a common prefix in long contexts, usually implemented at the provider or gateway level.
TTL (time-to-live)
An expiry value that determines how long a cached entry remains valid; after TTL passes, the entry is deleted or considered stale.
Cache key
A unique identifier generated from the inputs that determine a cache entry; for LLM calls, it typically combines the model, full system prompt, and user prompt.
Source: juejin.cn ↗ Google Translate ↗ Backup ↗