跪拜 Guibai
← Back to the summary

Exact Caching: The Same Question Asked 100 Times Now Costs One Answer

Series "AI Application Production Handbook" #15 (of 30) | Companion open-source project: https://github.com/ChenYingbo/ai-prod-demo This is Module 4 (Cost and Gateway) #2: C01 settled the accounts, C02 starts saving money — and it's the only "zero quality loss" cost-cutting method.

1. The Problem: The Books Are Right, But Money Still Leaks

C01 used cost_model.py to break down costs down to "how much each input/output token costs per call." After that, you'll likely see three phenomena:

  1. The same question is asked repeatedly — 10 users ask "What is RAG?" and the model dutifully answers 10 times, burning 10x the money.
  2. Hot questions eat most of the budget — after breaking down by model, you want to go further: which question types are burning money?
  3. Latency is high too — every repeated question waits for the model to regenerate, and users are stuck spinning.

One word solves it: caching. Identical inputs return the stored answer directly, not a single model call is made. Money saved, latency reduced, experience improved — a three-way win.

Core insight: Caching is the first lever in AI cost optimization because it's the only "zero quality loss" optimization — not swapping to a cheaper model (quality drops), but "not recomputing the same answer." But caching has two pitfalls: freshness and pollution. 90% of failures come from key design.

2. Principles: Three Cache Types, Don't Jump to the Most Complex First

2.1 Comparison of Three Mechanisms

Mechanism Hit Condition Implementation Cost Benefit Risk
Exact caching (this post) Input completely identical Low (hash key) High (hot questions get decent hit rate) Poor key design → hit rate ≈ 0
Semantic caching Input semantically similar Medium (embedding + vector search) Higher (paraphrases also hit) False hits (similar but shouldn't reuse)
Prefix Caching Same prefix (shared long context) Provider-side / gateway-side Reduces long-context cost Depends on provider support

Evolution path: start with exact caching (low cost, high determinism) → only when the hit rate truly can't improve, move to semantic caching → Prefix Caching basically depends on what the provider offers. Reversing the order is building your own incident.

2.2 Cache Key Design (The Lifeline of Hits)

key = hash(model + full system + full prompt)

All three elements must be present:

Anti-pattern: key contains only the prompt. The result: after gray release, old and new version answers cross-contaminate; after swapping models, all answers are stale from the old model.

2.3 Hit Rate and Benefit Calculation

hit rate = hits / (hits + misses)
savings ≈ hits × average cost per request (that's the avg_cost_per_request from C01)

First calculate "how much can be saved" then decide "whether it's worth doing." A system with only a 5% hit rate might be better off going back to optimize the prompt than implementing caching.

2.4 Invalidation Strategies

Strategy Method When to Use
TTL (this post) Cache expires after N seconds (CACHE_TTL_SECONDS) General default
Active invalidation Clear by key prefix when prompt/knowledge base updates Knowledge base Q&A (content changes must invalidate)
No invalidation TTL=0 Completely static content

Principle: better to over-invalidate than to over-hit. A stale cached answer is worse than no cache — users will take the old answer as truth.

3. Hands-On: From 6 Unit Tests to Real-Link Verification

3.1 Implementation (SQLite, 75 lines)

# app/cache.py core
@staticmethod
def make_key(model, system, prompt):
    raw = json.dumps({"m": model, "s": system, "p": prompt}, ensure_ascii=False)
    return hashlib.sha256(raw.encode("utf-8")).hexdigest()

def get(self, model, system, prompt):
    key = self.make_key(model, system, prompt)
    row = conn.execute("SELECT answer, created_at FROM llm_cache WHERE key=?", (key,)).fetchone()
    if not row: return None
    answer, created = row
    if self.ttl and time.time() - created > self.ttl:   # TTL expired, delete directly
        conn.execute("DELETE FROM llm_cache WHERE key=?", (key,)); return None
    conn.execute("UPDATE llm_cache SET hits = hits + 1 WHERE key=?", (key,))
    return answer

In production, swap SQLite for Redis without changing the interface a single line.

3.2 Unit Verification (No Key, No Network)

cd demo && source .venv/bin/activate
pytest tests/test_cache.py -q

Real output:

......                                                                   [100%]
6 passed in 1.50s

The 6 cases are: round-trip read/write / key distinguishes different inputs / TTL expiry / hit counting / chat_cached hit without calling LLM / statistics from logs.

3.3 Real-Link Verification: The Second Request Really Does Not Call the Model

LLM_BACKUP_MODEL=chat-cheap uvicorn app.main:app --port 8000
# In another terminal, send the same question twice
curl -X POST http://localhost:8000/api/chat -H "Content-Type: application/json" -d '{"question":"什么是 RAG?"}'
curl -X POST http://localhost:8000/api/chat -H "Content-Type: application/json" -d '{"question":"什么是 RAG?"}'

Real response (captured in the extreme scenario where the gateway is down, so the first request went through the degradation chain):

1st request (miss, went through degradation):

{"answer":"抱歉,模型服务暂时不可用,请稍后重试。","model":"rule-fallback","degraded":true,"from_cache":false}

2nd request (hit, same question):

{"answer":"抱歉,模型服务暂时不可用,请稍后重试。","model":"chat-cheap","degraded":false,"from_cache":true}

The corresponding latency from the logs tells the story:

event=cache_miss   latency_ms=11151   ← gateway down + fallback chain retries
event=cache_hit    latency_ms=3       ← hit, read straight from DB

11.1 seconds → 3 milliseconds. The 11 seconds here came from "gateway down but still stubbornly retrying"; normal generation is about 1-3 seconds. But the hit path is constant time — no queuing, no waiting on the model's mood.

3.4 Key Distinction Experiment (Do This Once)

from app.cache import ChatCache
c = ChatCache()
k1 = c.make_key('chat-strong', 'system-A', '什么是 RAG?')
k2 = c.make_key('chat-strong', 'system-B', '什么是 RAG?')
print('prompt changed, key changed:', k1 != k2)
print('completely identical, keys same:', k1 == c.make_key('chat-strong', 'system-A', '什么是 RAG?'))
prompt changed, key changed: True
completely identical, keys same: True

Feel this: including the full system in the key means a prompt upgrade automatically invalidates the old cache = no gray release cross-play. The leftover pitfall from R02 gets filled incidentally by C02.

3.5 Hit Rate and Savings Statistics

python scripts/cache_stats.py --log /tmp/app.log --avg-cost 0.000461
===== Cache Hit Statistics =====
hits 1 / misses 1 / total 2
hit rate: 50.0%
estimated savings: 1 × 0.000461 = 0.0005 yuan

0.000461 is the real average cost per request measured in C01 (10 calls, total cost 0.004608 yuan). Connect the cost model with caching, and "how much we saved" turns from a feeling into a number.

4. Real Pitfalls

  1. Key missing model/system — swapping models or changing prompts returns old answers; all three elements must be present
  2. Cache and gray release cross-play — canary users hit stable's old cache; including full system in the key isolates naturally
  3. No TTL set — users get old answers after knowledge base updates; content-type must use short TTL or active invalidation
  4. Hit rate not counted — you don't know if the cache is useful after implementing it; cache_hit/cache_miss events must be logged
  5. Cache privacy — sensitive question answers stored in plaintext (readable directly in SQLite); sensitive scenarios need encryption or skip caching
  6. Cache storm — hot keys expire all at once, all traffic goes back to source; add random jitter to TTL (same principle as A01's retry backoff)

5. What You Get From This Post

Next post (C03): Module 4 post 3, "Dynamic Routing and Model Selection" — simple questions go to cheap models, complex questions to expensive ones, turning C01's cost model into automatic decisions (with real routing tests). Follow + star, one post a day, 30 days to put AI applications into production.

Comments

Top 1 from juejin.cn, machine-translated. The original thread is authoritative.

HelloWorker

Like