Artificial Intelligence

KV Cache Is the Browser Cache for LLMs — and Prompt Caching Is the CDN

By 周末程序猿 · Aug 10, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

KV Cache is the single biggest reason LLM inference is fast enough to ship, and Prompt Caching is the single biggest lever for making it cheap enough to scale. A service that gets either wrong burns GPU memory on redundant computation, and at high concurrency the KV Cache alone can outgrow the model weights by 4×.

Summary

Every token an LLM generates would normally require recomputing attention over the entire sequence. KV Cache sidesteps that by keeping the Key and Value vectors of all processed tokens in memory, so each new decode step only computes attention for the newest token. The technique turns a cubic-cost problem into a linear one and is the reason autoregressive generation is practical at all. Without it, a 32-batch, 32k-context Qwen2.5-7B run would be impossible; with it, the KV Cache alone can still balloon to 57 GB — four times the model weights — which is why memory management matters as much as the math.

PagedAttention solves the resulting fragmentation by borrowing virtual-memory ideas: KV state is stored in fixed-size blocks, allocated on demand, and shared across requests that share a prefix. That block-level sharing is the foundation for Prompt Caching, where a long system prompt or tool definition is prefilled once and reused across every API call that starts with the same bytes. OpenAI and Anthropic both ship versions of this, with automatic prefix detection or explicit cache-control markers, and report up to 90% reductions in input cost and 85% drops in time-to-first-token.

The two layers form a complete caching stack: KV Cache makes a single generation fast, and Prompt Caching makes repeated generations cheap. The distinction is easy to miss because both store K and V vectors, but one operates inside a request and the other across requests. The Query vector is never cached — causal masking means a past Q is never queried again, so storing it would waste VRAM with no read path.

Takeaways

— KV Cache stores the Key and Value vectors of every processed token so the model never recomputes attention over the full sequence during decoding.

— Without KV Cache, generating L tokens costs roughly O(L³); with it, decode drops to O(N) per step and total cost approximates O(P² + P·L + L²).

— The Query vector is never cached because causal masking ensures a past Q is never queried again — storing it would waste VRAM with zero reads.

— In a Qwen2.5-7B FP16 deployment, a single 32k-token request uses about 0.45 GB of KV Cache; 32 concurrent 32k-token requests push that to 57 GB, roughly 4× the model weights.

— PagedAttention allocates KV state in fixed-size blocks instead of one contiguous chunk, cutting internal waste to under 4% and enabling prefix sharing, offloading, and block-level quantization.

— Prompt Caching reuses the KV state of a shared prefix across API calls, so the system prompt or tool definitions are prefilled only once.

— OpenAI automatically caches prefixes ≥1024 tokens at 128-token granularity; Anthropic requires explicit cache_control breakpoints and charges ~1.25× to write and ~0.1× to read.

— Combined, KV Cache and Prompt Caching can reduce time-to-first-token by up to 85% and input costs by up to 90% in long-prefix scenarios.

Conclusions

The KV Cache size scales with the product of batch size and sequence length, not with model size alone, which means concurrency is the real memory killer — not context length by itself.

Calling the technique 'KV Cache' obscures that it is really two distinct caches stacked: an intra-request compute cache and an inter-request prefix cache, each with different eviction policies and cost models.

PagedAttention is a direct lift from OS virtual memory, and the analogy holds all the way down: block tables, on-demand allocation, and external fragmentation are the same problems operating systems solved decades ago.

OpenAI's automatic prefix caching and Anthropic's explicit cache_control represent two philosophies — invisible infrastructure vs. developer-controlled cost knobs — and the tradeoff is between ease of use and predictability of billing.

Concepts & terms

KV Cache

A memory store that holds the Key and Value vectors computed for every token in a sequence. During autoregressive decoding, the model reads cached K/V for all past tokens and computes new K/V only for the latest token, avoiding O(N²) recomputation per step.

Prefill vs. Decode

The two phases of LLM inference. Prefill processes the entire input prompt in parallel, computes all K/V vectors, and stores them in the KV Cache (compute-bound). Decode generates one token at a time, reading from the cache and appending one new K/V pair per step (memory-bound).

PagedAttention

A memory management scheme for KV Cache that stores K/V vectors in fixed-size blocks instead of one contiguous allocation. It uses a block table to map logical token positions to physical blocks, reducing internal fragmentation to under 4% and enabling prefix sharing across requests.

Prompt Caching

A service-layer optimization that reuses the KV Cache state of a shared prefix (system prompt, tool definitions, conversation history) across multiple API calls. When a new request starts with the same bytes, the cached prefix is loaded directly, skipping Prefill for that portion and reducing input cost and latency.

Causal Masking

The attention mask that prevents a token from attending to future tokens. It is the reason Query vectors are never cached: a past Q is never queried by any future token, so storing it would consume memory with no possible access path.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗