The Prefill-Decode Split That Makes Every LLM Call Wasteful

A Brief Analysis of LLM Inference: Prefill and Decode

LLM inference sits at the intersection of system design and AI engineering. As models continue to upgrade and the efficiency of open-source and commercial models converges, LLM inference will enter a new phase starting in 2026, shifting from model training to model inference. Why? Statistics show that training compute accounts for only 30% of global usage, while inference accounts for 70%, a proportion that will continue to increase. This indicates that open-source models are already sufficient for most scenarios; the biggest problem now is how to popularize models and lower costs.

For this reason, I am shifting from summarizing last year's model training work toward the engineering practice of model inference. Having recently read some articles and books, I decided to write a handbook introducing the engineering practice of LLM inference. It is planned to cover inference across 12 chapters, including Prefill-and-Decode, KV cache, quantization, PagedAttention, FlashAttention, speculative decoding, parallelism strategies, vLLM, TensorRT-LLM, and SGLang.

Preface

Every time you send a message to ChatGPT, Claude, or Gemini, something remarkable happens within milliseconds. In some data center, a model with billions of parameters is awakened, reads your text, and begins generating a response—one token at a time, at a cost that would have seemed incredible five years ago.

Building an LLM requires a researcher's wisdom to construct the right architecture: where to use Attention, where to use Mamba layers, and what other layers to consider if building a hybrid model. Should you create an MoE (Mixture of Experts) model, or keep all weights active? These are research-level decisions.

Deploying and operating the model on a GPU cluster after it is built—so it can serve real user requests—falls into the engineering domain. Therefore, LLM inference is less a research discipline and more an engineering art.

Behind the word "inference" lies a complete engineering discipline that determines whether your AI product is fast or slow, cheap or expensive, scalable or fragile. Whether inference is done well determines if your product remains just an impressive demo or becomes a truly viable product.

Training a model happens once; inference happens billions of times a day. Every inefficiency accumulates, every wasted GPU cycle burns money, and every extra millisecond of latency loses users. This is why the world's top AI teams—Google, Meta, Anthropic, Mistral—all have entire engineering organizations dedicated to making inference faster, cheaper, and more scalable.

What are Prefill and Decode?

Current state-of-the-art models are decoder-based autoregressive models, meaning they predict the next token based on previous tokens. During the pre-training phase, the model learns to predict the next token based on the preceding text.

A token is the basic text or data unit that a Large Language Model (LLM) processes and understands. The model does not read text letter by letter; instead, it splits text into smaller chunks called tokens. A token can be a complete word, part of a word (subword), or a single punctuation mark.

When we send a prompt to an LLM, the model first processes the entire prompt in a single forward pass. This phase is called prefill. After reading the complete prompt, the model predicts the first output token. The time from sending the request to receiving the first generated token is called TTFT (Time to First Token).

Subsequently, the model enters the decode phase, generating one token at a time. The first generated token becomes part of the context, then the model predicts the second token. The first and second tokens become part of the context, then the model predicts the third token, and so on. This process continues until the model reaches the max_tokens limit or generates an EOS (End of Sequence) token.

Only after completing both the Prefill and Decode processes do we receive the final response from the LLM.

Example

Prompt:

The capital of France

The model first reads the complete prompt and predicts:

is

Now the context becomes:

The capital of France is

The model reads the complete context again and predicts the next token:

Paris

The context becomes:

The capital of France is Paris

Then the model predicts the next token:

Then predicts:

<EOS>

The final output is:

The capital of France is Paris.

Now, for every token, the attention layer in the model calculates K and V vectors.

A Concise Attention Example: Query, Key, and Value

Before diving deeper, let's quickly look at an example of how attention works and what Query (Q), Key (K), and Value (V) vectors are.

Before the model predicts the next token, each attention layer creates three vectors for every token:

Query (Q): What is this token looking for?
Key (K): What information does this token contain?
Value (V): What information should this token pass forward?

Let's use a very small prompt:

I love AI

Assume the model creates the following simple Key and Value vectors for each token:

Token	Key Vector K	Value Vector V
I	[1, 0]	[1, 0]
love	[0, 1]	[0, 2]
AI	[1, 1]	[3, 1]

Now, assume the current token is AI, and its Query vector is:

Q_AI = [1, 1]

The attention layer uses the dot product to compare this Query vector with the Key vector of each token.

Q × K^T, where K^T represents the transpose of the K vector

Q_AI · K_I    = [1, 1] · [1, 0] = 1
Q_AI · K_love = [1, 1] · [0, 1] = 1
Q_AI · K_AI   = [1, 1] · [1, 1] = 2

The raw attention scores are:

I     → 1
love  → 1
AI    → 2

These scores are converted to probabilities via the softmax function:

I     → 0.21
love  → 0.21
AI    → 0.58

This means the token AI is allocating attention as follows:

21% attention to "I"
21% attention to "love"
58% attention to "AI"

Now the model uses these attention weights to mix the Value vectors:

Attention Output =
0.21 × [1, 0] +
0.21 × [0, 2] +
0.58 × [3, 1]

The final output is approximately:

Attention Output ≈ [1.95, 1.00]

This final vector is the attention output for the token AI.

Simply put, attention allows the model to look at all relevant tokens, decide the importance of each token, and then combine their information.

Therefore, attention can be understood as:

Compare Query with Key → Get attention weights → Mix Values

This mixed information then continues to be passed through the next layers of the model, ultimately helping to predict the next token.

The Problem of Repeated Computation in the Decode Phase

Attention calculates K and V vectors for every token in the context.

Now imagine in the decode phase, as you know, we continuously add output tokens to the context. Therefore, for each new output token, we need to recalculate the K and V vectors for all previous tokens in the context.

Continuing with the example above, suppose we give the model this prompt:

The capital of France is

Assume this prompt has 5 tokens:

[The] [capital] [of] [France] [is]

In the first forward pass (prefill phase), the model calculates K and V vectors for all 5 input tokens:

Token 1: The      → K, V
Token 2: capital  → K, V
Token 3: of       → K, V
Token 4: France   → K, V
Token 5: is       → K, V

After this, the model predicts the first output token:

Paris

Prefill ends here, and Decode now begins.

The context becomes:

The capital of France is Paris

The model processes the complete context again, recalculating K and V vectors for all tokens:

Token 1: The      → K, V  Recalculated
Token 2: capital  → K, V  Recalculated
Token 3: of       → K, V  Recalculated
Token 4: France   → K, V  Recalculated
Token 5: is       → K, V  Recalculated
Token 6: Paris    → K, V  First calculation

Then the model predicts the next token:

The context becomes:

The capital of France is Paris.

The model recalculates K and V for the complete context again:

Token 1: The      → K, V  Recalculated
Token 2: capital  → K, V  Recalculated
Token 3: of       → K, V  Recalculated
Token 4: France   → K, V  Recalculated
Token 5: is       → K, V  Recalculated
Token 6: Paris    → K, V  Recalculated
Token 7: .        → K, V  First calculation

And predicts:

<EOS>

The K and V vectors for previous tokens remain unchanged, so recalculating them after each decode step is wasted work.

Thinking from a first-principles engineering perspective: how should this problem be solved?

If a computation is expensive and its results will be used repeatedly, what should we do?

Store them. The next chapter covers KV cache, and how to effectively solve the problem of repeated computation.

References

(1) https://pub.towardsai.net/llm-inference-handbook-2026-135c266b86e7 (2) https://handbook.modular.com/inference-optimization/prefill-decode-disaggregation/