Exact Caching: The Same Question Asked 100 Times Now Costs One Answer
Series "AI Application Production Handbook" #15 (of 30) | Companion open-source project: https://github.com/ChenYingbo/ai-prod-demo This is Module 4 (Cost and Gateway) #2: C01 settled the accounts, C02 starts saving money — and it's the only "zero quality loss" cost-cutting method.
1. The Problem: The Books Are Right, But Money Still Leaks
C01 used cost_model.py to break down costs down to "how much each input/output token costs per call." After that, you'll likely see three phenomena:
- The same question is asked repeatedly — 10 users ask "What is RAG?" and the model dutifully answers 10 times, burning 10x the money.
- Hot questions eat most of the budget — after breaking down by model, you want to go further: which question types are burning money?
- Latency is high too — every repeated question waits for the model to regenerate, and users are stuck spinning.
One word solves it: caching. Identical inputs return the stored answer directly, not a single model call is made. Money saved, latency reduced, experience improved — a three-way win.
Core insight: Caching is the first lever in AI cost optimization because it's the only "zero quality loss" optimization — not swapping to a cheaper model (quality drops), but "not recomputing the same answer." But caching has two pitfalls: freshness and pollution. 90% of failures come from key design.
2. Principles: Three Cache Types, Don't Jump to the Most Complex First
2.1 Comparison of Three Mechanisms
| Mechanism | Hit Condition | Implementation Cost | Benefit | Risk |
|---|---|---|---|---|
| Exact caching (this post) | Input completely identical | Low (hash key) | High (hot questions get decent hit rate) | Poor key design → hit rate ≈ 0 |
| Semantic caching | Input semantically similar | Medium (embedding + vector search) | Higher (paraphrases also hit) | False hits (similar but shouldn't reuse) |
| Prefix Caching | Same prefix (shared long context) | Provider-side / gateway-side | Reduces long-context cost | Depends on provider support |
Evolution path: start with exact caching (low cost, high determinism) → only when the hit rate truly can't improve, move to semantic caching → Prefix Caching basically depends on what the provider offers. Reversing the order is building your own incident.
2.2 Cache Key Design (The Lifeline of Hits)
key = hash(model + full system + full prompt)
All three elements must be present:
- model: different models give different answers; the cache can't be shared.
- system (full prompt text): if the prompt changes version, old cache must auto-invalidate — this step incidentally solves the R02 gray release cross-play problem (canary uses different prompt = different key = naturally isolated cache)
- prompt: the question itself
Anti-pattern: key contains only the prompt. The result: after gray release, old and new version answers cross-contaminate; after swapping models, all answers are stale from the old model.
2.3 Hit Rate and Benefit Calculation
hit rate = hits / (hits + misses)
savings ≈ hits × average cost per request (that's the avg_cost_per_request from C01)
First calculate "how much can be saved" then decide "whether it's worth doing." A system with only a 5% hit rate might be better off going back to optimize the prompt than implementing caching.
2.4 Invalidation Strategies
| Strategy | Method | When to Use |
|---|---|---|
| TTL (this post) | Cache expires after N seconds (CACHE_TTL_SECONDS) |
General default |
| Active invalidation | Clear by key prefix when prompt/knowledge base updates | Knowledge base Q&A (content changes must invalidate) |
| No invalidation | TTL=0 | Completely static content |
Principle: better to over-invalidate than to over-hit. A stale cached answer is worse than no cache — users will take the old answer as truth.
3. Hands-On: From 6 Unit Tests to Real-Link Verification
3.1 Implementation (SQLite, 75 lines)
# app/cache.py core
@staticmethod
def make_key(model, system, prompt):
raw = json.dumps({"m": model, "s": system, "p": prompt}, ensure_ascii=False)
return hashlib.sha256(raw.encode("utf-8")).hexdigest()
def get(self, model, system, prompt):
key = self.make_key(model, system, prompt)
row = conn.execute("SELECT answer, created_at FROM llm_cache WHERE key=?", (key,)).fetchone()
if not row: return None
answer, created = row
if self.ttl and time.time() - created > self.ttl: # TTL expired, delete directly
conn.execute("DELETE FROM llm_cache WHERE key=?", (key,)); return None
conn.execute("UPDATE llm_cache SET hits = hits + 1 WHERE key=?", (key,))
return answer
In production, swap SQLite for Redis without changing the interface a single line.
3.2 Unit Verification (No Key, No Network)
cd demo && source .venv/bin/activate
pytest tests/test_cache.py -q
Real output:
...... [100%]
6 passed in 1.50s
The 6 cases are: round-trip read/write / key distinguishes different inputs / TTL expiry / hit counting / chat_cached hit without calling LLM / statistics from logs.
3.3 Real-Link Verification: The Second Request Really Does Not Call the Model
LLM_BACKUP_MODEL=chat-cheap uvicorn app.main:app --port 8000
# In another terminal, send the same question twice
curl -X POST http://localhost:8000/api/chat -H "Content-Type: application/json" -d '{"question":"什么是 RAG?"}'
curl -X POST http://localhost:8000/api/chat -H "Content-Type: application/json" -d '{"question":"什么是 RAG?"}'
Real response (captured in the extreme scenario where the gateway is down, so the first request went through the degradation chain):
1st request (miss, went through degradation):
{"answer":"抱歉,模型服务暂时不可用,请稍后重试。","model":"rule-fallback","degraded":true,"from_cache":false}
2nd request (hit, same question):
{"answer":"抱歉,模型服务暂时不可用,请稍后重试。","model":"chat-cheap","degraded":false,"from_cache":true}
The corresponding latency from the logs tells the story:
event=cache_miss latency_ms=11151 ← gateway down + fallback chain retries
event=cache_hit latency_ms=3 ← hit, read straight from DB
11.1 seconds → 3 milliseconds. The 11 seconds here came from "gateway down but still stubbornly retrying"; normal generation is about 1-3 seconds. But the hit path is constant time — no queuing, no waiting on the model's mood.
3.4 Key Distinction Experiment (Do This Once)
from app.cache import ChatCache
c = ChatCache()
k1 = c.make_key('chat-strong', 'system-A', '什么是 RAG?')
k2 = c.make_key('chat-strong', 'system-B', '什么是 RAG?')
print('prompt changed, key changed:', k1 != k2)
print('completely identical, keys same:', k1 == c.make_key('chat-strong', 'system-A', '什么是 RAG?'))
prompt changed, key changed: True
completely identical, keys same: True
Feel this: including the full system in the key means a prompt upgrade automatically invalidates the old cache = no gray release cross-play. The leftover pitfall from R02 gets filled incidentally by C02.
3.5 Hit Rate and Savings Statistics
python scripts/cache_stats.py --log /tmp/app.log --avg-cost 0.000461
===== Cache Hit Statistics =====
hits 1 / misses 1 / total 2
hit rate: 50.0%
estimated savings: 1 × 0.000461 = 0.0005 yuan
0.000461 is the real average cost per request measured in C01 (10 calls, total cost 0.004608 yuan). Connect the cost model with caching, and "how much we saved" turns from a feeling into a number.
4. Real Pitfalls
- Key missing model/system — swapping models or changing prompts returns old answers; all three elements must be present
- Cache and gray release cross-play — canary users hit stable's old cache; including full system in the key isolates naturally
- No TTL set — users get old answers after knowledge base updates; content-type must use short TTL or active invalidation
- Hit rate not counted — you don't know if the cache is useful after implementing it;
cache_hit/cache_missevents must be logged - Cache privacy — sensitive question answers stored in plaintext (readable directly in SQLite); sensitive scenarios need encryption or skip caching
- Cache storm — hot keys expire all at once, all traffic goes back to source; add random jitter to TTL (same principle as A01's retry backoff)
5. What You Get From This Post
- Can compare exact / semantic / prefix three cache types, clarify the evolution order
- Can state the three key elements of the cache key and that anti-pattern
- Can write the hit rate and savings formulas, connected to C01's avg_cost
-
pytest tests/test_cache.py6/6 passed - Real link shows
from_cache=false→from_cache=true - Done the key experiment: when the prompt upgrades, old cache auto-invalidates
Next post (C03): Module 4 post 3, "Dynamic Routing and Model Selection" — simple questions go to cheap models, complex questions to expensive ones, turning C01's cost model into automatic decisions (with real routing tests). Follow + star, one post a day, 30 days to put AI applications into production.
Top 1 from juejin.cn, machine-translated. The original thread is authoritative.
Like