← Back to the summary

KV Cache Is the Browser Cache for LLMs — and Prompt Caching Is the CDN

Imagine a user opens a website with a logo, CSS files, JavaScript files, fonts, and images. Without caching, every time the user visits a new page, the browser would have to re-download the same logo, the same CSS, the same JavaScript, and the same fonts from the server, which would be a huge waste.

The logo hasn't changed, the CSS file hasn't changed, the JavaScript file hasn't changed — why re-download them?

Browsers use caching to solve this problem, and the same principle applies here: we store the computed K and V vectors in a cache. During the Prefill phase, when the model works hard to compute the first token, since it already has access to the entire input prompt, it caches the K and V vectors of the prompt tokens.

KV cache is like a browser cache, but instead of caching images, CSS, and JavaScript files, it caches the Key and Value vectors of previous tokens.

This avoids redundant computation, making token-by-token generation much faster.

Example

Let's understand this with the same prompt:

The capital of France is

In the first forward pass, the model computes K and V vectors for all 5 tokens:

The      → K1, V1
capital  → K2, V2
of       → K3, V3
France   → K4, V4
is       → K5, V5

The model stores these K and V vectors in memory.

This stored memory is called the KV cache.

Now the model predicts the first output token:

Paris

Thanks to the KV cache:

The      → use cached K1, V1
capital  → use cached K2, V2
of       → use cached K3, V3
France   → use cached K4, V4
is       → use cached K5, V5
Paris    → compute new K6, V6

The model no longer recomputes K and V for all 6 tokens; it only needs to compute K and V for the new token.

Then the model predicts the next token:

The context becomes:

The capital of France is Paris.

Only the new token . requires new K and V computation:

The      → use cached K1, V1
capital  → use cached K2, V2
of       → use cached K3, V3
France   → use cached K4, V4
is       → use cached K5, V5
Paris    → use cached K6, V6
.        → compute new K7, V7

Compare this to the previous example: without KV cache, every output token requires recomputing K and V vectors for all tokens; with KV cache, we only need to compute K and V vectors for the output token.

Computational Characteristics of Prefill and Decode

In the Prefill phase, we compute K and V vectors and store them in the KV cache, so this operation is compute-bound.

In the Decode phase, we read values from the KV cache far more frequently than we compute K and V vectors for tokens.

Assuming an output of 6 tokens, for 5 of those tokens we read K and V vectors from the KV cache, and only compute vectors for the 6th token. Therefore, this operation is memory-bound.

Why Is It Called KV Cache? — Where Did Q Go?

Everyone new to KV Cache asks: if K and V can be cached, what about Q?

Analogy: Q is the "question posed by the current token," K/V are the "index cards + content summaries of historical tokens." The second question doesn't need to "remember the first question," it only needs the same set of cards and bookshelf.

Role	Meaning	Lifecycle
Q (Query)	"What should I pay attention to?"	One-time: discarded after use
K (Key)	"What index am I?"	Persistent: every subsequent token queries it
V (Value)	"What do you get by attending to me?"	Persistent: every subsequent token reads it

At Decode step t:

Q_t needs to query: K_1..K_t      ← needs all historical Keys
Q_t needs to read:  V_1..V_t      ← needs all historical Values
Q_t does not need: Q_1..Q_{t-1}  ← past Queries are meaningless for new tokens

Causal masking determines Q's one-time nature: Q_{t+1}'s query range is [1, t+1], and it will never again use Q_t to query others. Therefore: caching something that will never be accessed again is a pure waste of VRAM.

Scale intuition (LLaMA-2 70B, GQA, seq=4096):

K + V ≈ 1.25 GB
If Q were forcibly stored (64 Q heads) ≈ +5 GB, and these 5 GB would never be read

Prefill / Decode: How KV Cache Reduces Computation

1. Reduction in Computation

Prefill: Input the entire prompt, compute K/V for all tokens in parallel, write to cache, produce the first output token. Decode: At each step, input only the new token from the previous step, compute its Q/K/V; append the new K/V into the cache; use "historical K/V + current K/V" to compute attention, then output the next token.

Without KV Cache: Single-step attention is approximately O(N²); the cumulative cost of generating L_gen tokens is approximately O(L_gen³). With KV Cache: Decode single-step drops to O(N); total cost is approximately O(P² + P·L_gen + L_gen²) (P is prompt length, P² comes from Prefill).

Simplified pseudocode:

class KVCache:
    def __init__(self):
        self.cache = {"key": None, "value": None}

    def update(self, key, value):
        if self.cache["key"] is None:
            self.cache["key"] = key
            self.cache["value"] = value
        else:
            # Concatenate along the seq dimension; if layout is [B, heads, seq, dim], change dim=2
            self.cache["key"] = torch.cat([self.cache["key"], key], dim=1)
            self.cache["value"] = torch.cat([self.cache["value"], value], dim=1)

2. Reduction in VRAM Usage

Memory_KV ≈ 2 × b_kv × L × B × S × H × (N_kv / N_attn)

2: One copy each for K and V
b_kv: Bytes (FP16=2)
L: Number of layers; B: Concurrent requests; S: Average sequence length
H: Hidden size; N_kv / N_attn: KV heads / Q heads (<1 for GQA)

Key point: S × B is a multiplicative relationship. Long context × high concurrency can push KV Cache to exceed the model weights.

Taking Qwen2.5-7B (FP16) as an example, approximately 56 KB per token:

batch	seq_len	KV Cache	vs Weights (~14 GB)
1	2,048	0.11 GB	<1%
8	32,768	14.3 GB	≈ weights
32	32,768	57 GB	≈ 4× weights

3. PagedAttention: How to Manage KV VRAM Fragmentation

Problem: Contiguously pre-allocating KV for each request based on max_model_len →

Reservation waste: Allocated 32K, but only generated 3K
External fragmentation: Holes released by short requests cannot be combined into the next large contiguous block

Traditional solutions often achieve only 20–40% effective KV utilization.

Solution (analogous to OS virtual memory):

OS	PagedAttention
Physical page frames	KV block (fixed block_size tokens)
Page table	Block Table (logical → physical)
MMU hardware translation	Attention kernel software table lookup gather
On-demand paging	Blocks allocated on demand, recycled after use

Result: Waste is compressed to <4% (mainly the last block of each request being partially filled), and throughput often increases 2–4× for the same VRAM capacity.

Unlocked by extension:

Prefix Caching: Multiple requests' block tables can point to the same physical block (shared identical prefixes)
Offloading: Swap in/out at block granularity, more flexible than moving entire contiguous large tensors
Quantization: Apply FP8/INT8 within blocks, with dequantization on the kernel side

Prompt Caching: "Prefix KV Reuse" Across Requests

The KV Cache discussed earlier solves the problem of not recomputing history within the same generation. In real products, there is an even more "cost-saving" layer of caching: sharing the same prefix across multiple API calls.

1. What is Prompt Caching

Keep the KV (or equivalent intermediate state) computed from a long and stable prefix (system prompt, tool definitions, knowledge base, previous conversation turns) on the server side; if the prefix of the next request is byte-for-byte identical, reuse it directly, and only perform Prefill for the changed suffix.

2. Prompt Caching Methods Provided by OpenAI and Anthropic

OpenAI

Public prefixes ≥1024 tokens automatically participate in caching, with hit granularity incrementing at 128 tokens
Hits significantly reduce TTFT and input costs (official example: TTFT reduced by up to ~80%, input costs by up to ~90%)
Key engineering discipline: Place static content at the front, put volatile content like timestamps in metadata; tool/example ordering must also be stable

Claude

Use cache_control to explicitly mark cache breakpoints (prefix byte-level matching)
Fixed order: tools → system → messages; if the front part changes, everything after it becomes invalid
Cost: Writing cache is about 1.25× the input price, reading cache is about 0.1×; official long-prompt scenarios can reduce costs by ~90% and TTFT by ~85%

Self-hosted (vLLM)

Implements Automatic Prefix Caching on top of PagedAttention: blocks with the same prefix are directly reused, naturally saving Prefill.

3. KV Cache ≠ Prompt Caching (Comparison Summary)

	KV Cache	Prompt Caching / Prefix Cache
Scope	Within a single request: token 1→N output	Across requests: multiple calls share the same prefix
What is stored	K, V from each layer's Attention	KV / intermediate state corresponding to the prefix (reused by prefix hash)
Problem solved	Decode doesn't recompute history, `O(N²) → O(N)`	Prefill for repeated prefixes is paid only once, reducing TTFT/cost
User controllable	Framework automatic (e.g., `use_cache=True`)	OpenAI automatic + `prompt_cache_key`; Claude requires `cache_control`
Typical bottleneck	VRAM capacity + HBM bandwidth	Prefix stability, routing to the same cache, TTL
Relationship	Is the infrastructure	Is an extension of the KV Cache concept at the "service layer"

4. Summary

KV Cache prevents "this generation" from recomputing; Prompt Caching prevents "the next request" from redoing the Prefill of the same prefix. Only when combined do they form the complete caching stack that makes modern LLM services "fast and cheap."