The Prefill-Decode Split That Makes Every LLM Call Wasteful
Inference already eats 70% of global AI compute and the ratio keeps growing. Every redundant K-V recalculation in the decode phase burns GPU cycles and adds latency that compounds across billions of daily calls, so eliminating that waste is the difference between a product that scales and one that bleeds money.
Every autoregressive LLM call runs through a prefill phase that processes the full prompt in one forward pass, then a decode phase that generates output tokens sequentially. The split creates a fundamental inefficiency: during decode, the model recomputes Key and Value vectors for every token already in the context, even though those vectors never change. A walkthrough with a simple prompt shows how the attention mechanism compares Query vectors against Keys to produce attention weights, then mixes Value vectors to predict the next token. The repeated K-V computation in decode is pure waste — the vectors are identical each time. The engineering answer is to cache them, which is what KV cache solves. Inference now consumes 70% of global AI compute and that share is still climbing, making this optimization a cost-of-business problem, not a research curiosity.
The 70/30 inference-to-training compute split reframes the entire AI cost conversation: model quality is no longer the binding constraint, inference economics is.
Prefill and decode have fundamentally different compute profiles — prefill is parallel and compute-bound, decode is sequential and memory-bound — which means optimizing one often hurts the other.
The article's framing of inference as an engineering discipline rather than a research problem reflects a real industry shift as open-source models close the capability gap with commercial ones.