Cloud Native

Inside the Inference Framework: Why Output Costs 4x Input, and Agent Workloads Changed the Math

By 阿里云云原生 · Oct 22, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

API bills and self-hosted latency are decided in this layer: cache hit rate, TTFT, token throughput, and PD separation all show up as either price or waiting time. Teams that build agents should treat prompt layout and framework version as cost controls, not formatting details.

Summary

An inference framework sits between the model and the GPU. It loads weights, schedules requests, manages KV Cache, and returns tokens. Because a model writes one output token at a time during decode and reads all weights for every token, output tokens cost several times input tokens; because cached prefixes skip prefill, cache hits are dramatically cheaper. Agent requests are long, repetitive, and short in output, so the framework's job shifts from pure speed to reuse: remember what has already been computed and route the next request to the machine that has it.

The open-source landscape has converged on continuous batching, paged KV Cache, prefix caching, chunked prefill, speculative decoding, and structured output. Differentiation has moved to support speed for new models, hardware coverage, and cluster-level scheduling: PD separation, cross-machine KV Cache, and autoscaling. DeepSeek's path from vLLM fork to DeepInfer, plus vLLM supporting V3.2-Exp on release day, shows model and framework are now co-designed.

Takeaways

— Decode is the expensive half of generation: one token at a time, with all model weights read from memory each step, so output tokens are priced higher than input tokens.

— Using DeepSeek pricing as an example, output tokens cost 4 times input tokens and cache-hit input tokens cost 1/50 of cache-miss input tokens.

— Continuous batching is now standard in mainstream inference engines; completed requests leave the batch and new requests join at every decode step.

— KV Cache grows with context length and concurrency, and its capacity often decides how many requests a single GPU can serve.

— Prefix caching reuses the KV Cache of unchanged leading content, which is why stable system prompts and tool definitions at the start of a request are a direct cost lever for agent developers.

— PagedAttention, introduced by vLLM, cuts KV Cache memory waste by allocating fixed-size pages instead of reserving a contiguous block per request.

— Core engine features have converged: continuous batching, paged KV Cache, prefix caching, chunked prefill, speculative decoding, and structured output are common across vLLM, SGLang, TensorRT-LLM, and LMDeploy.

— As of October 2026, GitHub stars are roughly 94k for vLLM, 37k for SGLang, 15k for TensorRT-LLM, 8k for LMDeploy, over 130k for llama.cpp, and over 180k for Ollama.

— TGI entered maintenance mode in December 2025 and directs users to vLLM, SGLang, and similar engines.

— Model release and framework support are increasingly synchronized: vLLM supported DeepSeek V3.2-Exp on release day and Qwen3-Next the next day.

— DeepSeek-V3 has 671 billion total parameters but activates about 37 billion per token, so all experts must stay resident in memory and expert parallelism is required.

— PD separation runs Prefill and Decode on different GPU groups so each phase can be optimized for its own bottleneck: compute for Prefill, memory bandwidth for Decode.

— Speculative decoding uses a small draft model to propose several tokens and the large model to verify them in parallel, preserving the output distribution while speeding up generation.

— Structured output compiles format rules into a grammar and excludes illegal tokens during sampling, guaranteeing valid JSON for tool calls.

Conclusions

When the same GPUs run at double inference efficiency, an API provider can serve twice the users or halve the price; that arithmetic makes inference frameworks the real competitive surface of model API pricing.

Agent contexts make every request a repetition of the last one, so cache hit rate becomes a direct cost line; a single changing timestamp at the start of a system prompt can invalidate the whole prefix.

The fast path from papers like Orca, PagedAttention, FlashAttention, and DistServe into mainstream engines explains why core features now look identical, leaving release-day model support and cluster behavior as the differentiators.

DeepSeek building its internal engine on an early vLLM branch, then open-sourcing components instead of the fork, shows that a tightly coupled production stack is hard to extract even when a company is committed to open source.

Hardware diversity is expanding because model vendors are designing numeric formats for upcoming domestic chips, yet each framework-chip-model combination still needs separate validation, so portability remains a roadmap item rather than an automatic property.

Concepts & terms

Inference framework

Middleware between a trained model and GPU hardware that loads weights, accepts requests, schedules computation, manages KV Cache, and returns generated tokens. It determines first-token latency, throughput, cost, stability, and output accuracy.

Prefill and Decode

The two phases of generation. Prefill reads all input tokens in parallel and produces the first output token, and is compute-bound. Decode then produces one output token at a time and is limited by memory bandwidth because model weights are read from memory each step.

KV Cache

Saved intermediate Key/Value results from attention, used to avoid recomputing earlier context. Its memory footprint grows with context length and concurrency, and it often decides how many requests one GPU can serve.

Continuous Batching

A scheduling technique where completed requests leave a batch and waiting requests join at each decode step, so the GPU stays busy without waiting for the slowest request to finish.

Prefix Caching

Reusing the KV Cache of unchanged leading tokens such as system prompts, tool definitions, and earlier turns, so only new tokens need Prefill. It is the mechanism behind low cache-hit prices on model APIs.

PD Separation

Deploying Prefill and Decode on separate groups of GPUs because they have different bottlenecks: compute versus memory bandwidth. The Prefill side passes KV Cache to the Decode side over a fast network.

Speculative Decoding

Using a small draft model to propose several future tokens, then having the large model verify them in parallel. It speeds up generation while preserving the same output distribution as normal sampling.

MoE (Mixture of Experts)

A model architecture with many expert sub-networks where each token activates only a few experts, expanding knowledge capacity without adding per-token compute. All experts must stay resident in memory, so MoE models usually require expert parallelism across GPUs.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗