Inference Frameworks for Non-Algorithm Engineers

Author: Wang Chen

Following Model and Harness, DeepSeek has open-sourced DeepInfer, an inference framework that adds another important piece to help enterprises completely master the agent progress bar and pushes the term inference framework into public view.

Although Model and Harness are now familiar concepts, inference framework, a term used mainly by algorithm engineers, still feels unfamiliar. It is closely tied to the cost and experience of using models, for example why the output price is 4 times the input price, and why cache hit price is 1/50 of the miss price (using DeepSeek's current pricing as an example). This article should provide some easy-to-understand answers to these questions.

This article is divided into five chapters.

• What an inference framework is, and how it relates to models and agents.
• Several core concepts needed to read further.
• The current state and development trends of mainstream open-source inference frameworks.
• Starting from one request, how an inference framework works.
• Which groups and enterprises use inference frameworks.

The article is long. You can also get a better reading experience here: https://aliyun.github.io/ai-agent-handbook/optimization/chapter-17/inference.html

Models and Inference Frameworks

1. The life of a model: from training to launch

A large model goes through roughly four stages from nothing to ready. First is pretraining, where massive text corpora teach the model language and knowledge. Second is post-training, where supervised fine-tuning, reinforcement learning, distillation, and other methods teach the model to follow instructions, use tools, and complete specific tasks. Third is evaluation, which confirms the model meets the target task performance. Fourth is deployment, where the model is called by applications as an API or a private service, with continuous monitoring and iteration in production.

Inference means running the model: using the trained model to compute an output from an input. An inference framework is responsible for doing this well: loading the model onto various hardware and software environments, receiving requests, organizing computation, and returning results.

A prerequisite is to make the model run at all, meaning support for the model to be deployed and the hardware in use, including compatibility. That is the basic job of any framework technology.

On that basis, 'good' has five aspects.

• Fast: the user sees the first character as soon as possible after sending a request, and generation is smooth without stalling.
• Many: the same hardware can serve as many requests as possible at the same time.
• Economical: use memory and compute fully, avoid recomputing what has already been computed, and spread the cost of every call.
• Stable: when traffic peaks, extremely long requests, or hardware failures occur, the service remains available and scales with business size.
• Accurate: output after deployment stays consistent with the model's original capability, meets format requirements, and can be reproduced when needed.

These five aspects often constrain each other: packing in more requests improves 'many' but slows the generation speed of each request; lowering numeric precision improves 'economical' but can hurt 'accurate'. The inference framework's job is to find a balance among them according to business needs.

Most concepts and designs described later can be traced to one or more of the five: time to first token and generation speed map to fast; batching and throughput map to many; memory management, quantization and caching map to economical; cluster scheduling and fault tolerance map to stable; chat templates, sampling and structured output map to accurate.

In the lifetime of a model from training to launch, an inference framework appears beyond just the launch stage. At launch, every user question and every agent model call runs through an inference framework. In post-training it is also essential: reinforcement learning requires the model to repeatedly try tasks and generate many trajectories, then update weights based on results. The generation step is often the most time-consuming part of the whole training pipeline, so mainstream RL training frameworks generally embed mature inference engines dedicated to batch generation. Evaluation requires running the model through thousands of questions, and synthetic data requires the model to produce samples in batches; these also use inference frameworks.

There is a fundamental cost-structure difference between training and inference. Training is a one-time investment: a model trains for weeks or months and then stops. Inference is a continuing expense: every call after launch consumes compute, and the larger the call volume, the longer the bill.

2. The more complex the task, the higher the requirements on the inference framework

If a chatbot is one question and one answer, an agent executing a task is more like a continuous multi-turn dialogue. Take an agent that helps you troubleshoot an online failure: it first understands the problem, makes a plan, calls a log-query tool, reads the returned result, decides the next step, calls a monitoring tool, decides again... In one task, the model may be called a dozen or even dozens of times. Fault diagnosis in Alibaba Cloud's full-domain operations intelligence platform STAROps is exactly this kind of scenario: the user describes a problem in natural language, and the agent narrows down the scope through multi-turn dialogue, cross-analyzing logs, metrics, traces, changes, and other data.

Moreover, each call sends the system prompt, tools, previous reasoning from every earlier step, and tool return results, plus new content, to the model. As the task progresses, the context gets longer, and most of each request has already appeared in the previous request.

This gives inference load three characteristics: very long input, relatively short output, and high repetition between consecutive requests. At the same time, the end-to-end latency of an agent is the sum of each step's latency; every call that waits one extra second becomes dozens of extra seconds over dozens of steps.

This creates new requirements for the inference framework. In chat scenarios, frameworks mainly compete on who computes faster; in agent scenarios, they must also compete on who computes less. Whether already-computed content can be remembered and reused directly determines inference cost and speed.

3. Training frameworks vs. inference frameworks

Many engineers instinctively compare inference frameworks with training frameworks such as PyTorch, Megatron-LM, and DeepSpeed. Both deal with models and GPUs, but the problems they solve are completely different.

Comparison dimension	Training framework	Inference framework
Core goal	Train a model	Use a model
Computation	Forward plus backward propagation, updating weights	Forward only, fixed weights
Load characteristics	Data prepared in batches, runs according to plan	Requests arrive randomly, vary in length, enter and leave at any time
Time requirement	Training for days or weeks is acceptable	Each request must respond in seconds or less
Main use of GPU memory	Weights, gradients, optimizer states, intermediate activations	Weights and KV Cache (explained later)
Focus metrics	Training throughput, convergence, compute utilization	Time to first token, generation speed, throughput, unit cost

This table shows that an inference framework is more like an online serving system. The problems it faces are similar to those of databases and web servers: request queueing, resource scheduling, cache hits, load balancing, and failure recovery. Only the resources being scheduled are not CPU and disk but expensive GPUs and GPU memory.

4. Inference frameworks, models, agents, and Harness

These familiar concepts can be used to understand the inference framework's place.

In an agent application, Harness is responsible for organizing prompts, managing context, and calling tools. Below it is the model service, which can be called via API or self-deployment. Below model service is the inference framework, the middleware layer between the model and the hardware environment after launch. Below the inference framework are the GPUs and other hardware.

If you use models through APIs and subscriptions in daily work, you usually do not notice the inference framework. But enterprises that deploy models themselves will find that a good inference framework decides the cost and experience: inference cost, how long the first token takes, how many tokens per second, and whether things slow down under concurrency are all decided at this layer.

Core Concepts Needed for the Rest of This Article

1. Several easily confused terms

Articles about inference often show terms like inference engine, inference framework, inference service, and model serving platform. They are often used interchangeably.

Term	What it refers to	Typical form
Inference engine	The core program that runs a model on one or a few machines, responsible for request scheduling, memory management, and computation execution	A program or Python library that can be started on a server
Inference framework	Often used interchangeably with inference engine in daily usage; the fuller meaning also includes API access, distributed deployment, caching systems, and other supporting components	An open-source software suite, usually with an inference engine at its core
Inference service	An inference framework deployed as an online service, with gateway, authentication, billing, monitoring, and scaling added	An enterprise internal model service cluster
Model serving platform (MaaS)	Cloud vendors turn inference services into products; users pay per call and do not need to care about the underlying layer	Model APIs from cloud vendors and model vendors
Local-run tool	Packs an inference engine, model download, and a simple API together for personal computers	A desktop tool that runs a model on a laptop with one command

A simple way to think about it is layer by layer: the inference engine is the heart, the inference framework is the whole power system containing the heart, the inference service is a fleet ready to operate on the road, and the model serving platform is a passenger transport company selling tickets to the public. The rest of this article uses the term inference framework uniformly.

2. Token: the smallest unit a model reads and writes

A model does not process text directly. It first splits text into tokens and then converts them into numeric IDs. In English, one token is roughly three-quarters of a word; in Chinese, the number of tokens per character depends on the model's tokenizer.

Tokens are also the billing unit for large models. Almost all model APIs price input tokens and output tokens separately, and output tokens usually cost several times more than input tokens. Why is writing more expensive than reading? The answer is in the next concept.

3. Prefill and Decode: read the question first, then answer

A model processes a request in two phases.

The first phase is Prefill, roughly 'reading the question'. The model reads all input at once, all input tokens are computed in parallel, and the GPU's compute units are fully utilized. At the end of this phase, the model produces the first output token.

The second phase is Decode, roughly 'answering'. The model can only generate one token at a time. After generating one, it appends that token to the existing content and generates the next, repeating until it finishes. Each generated token requires reading all model weights from GPU memory while doing very little computation. This is like flipping through an entire reference book to write one character. Therefore the bottleneck in Decode is not compute capability but memory bandwidth, the speed of 'flipping the book'.

The two phases have completely different characteristics: Prefill is compute-hungry and Decode is bandwidth-hungry. This also answers the earlier question: output is more expensive because it must be generated serially one token at a time, occupying the GPU longer per unit of output.

Since the biggest Decode cost is 'flipping the book', can one flip serve many requests at once? This is batching: multiple requests are grouped into a batch and share the same weight read. Processing 1 request or 64 requests takes similar time for one Decode step, but the number of tokens produced differs by dozens of times. Early batching waited until a full batch was ready and did not switch to the next batch until the slowest request finished. Later, Continuous Batching schedules at the granularity of each generation step: finished requests leave at any time, new requests join at any time, like a loop bus that people can get on and off freely. Today, Continuous Batching is standard in mainstream inference frameworks.

4. KV Cache: the model's scratch paper

When a large model generates each new token, it must 'look back' at all previous content to decide which information is most relevant now. This is the attention mechanism. If every token regenerated all previous content from scratch, computation would explode as length grows.

The solution is to save the intermediate results from computing each previous token (called Key and Value) and reuse them when generating a new token; only the new token itself needs to be computed. These saved intermediate results are the KV Cache. It is like the model's scratch paper: whatever was computed before is written down, and later work just looks it up.

The scratch paper is not free. KV Cache size is proportional to context length and proportional to the number of concurrently processed requests. The longer the context and the higher the concurrency, the thicker the scratch paper and the more GPU memory it occupies. In real inference services, KV Cache is often the most constrained memory resource, and its capacity directly decides how many requests one GPU can serve concurrently.

Quantization is a common way to make the scratch paper and the model itself 'thinner': store weights, KV Cache, or intermediate results in lower numeric precision, trading a little accuracy for large memory savings and speed gains. Dropping from 16-bit to 8-bit has little effect on most general tasks and is a common production choice. Dropping to 4-bit or lower compresses more, but the effect depends on the specific model and task and must be validated with your own business data. In addition, many new models are designed with attention architectures that compress KV Cache, making the scratch paper thinner at the source. Whether a model can be deployed depends first on the memory math of its weights; how many people it can serve depends mainly on the memory math of its KV Cache.

5. TTFT, TBT, and throughput: three indicators that trade off against each other

The most common indicators for inference performance are three.

• TTFT (Time To First Token): the time from sending a request to receiving the first output token, or how long the user waits before the model 'speaks'. It is mainly decided by the Prefill phase.
• TBT (Time Between Tokens), also called TPOT (Time Per Output Token): the average interval between two consecutive output tokens. It decides the model's speaking speed and is mainly decided by the Decode phase.
• Throughput: the number of tokens or requests the whole system processes per unit time. It decides how many users a set of hardware can serve and directly decides unit cost.

These three indicators are very hard to maximize at the same time. A larger batch raises throughput but gives each request a smaller time slice and slows generation. Prioritizing the Prefill of new requests reduces time to first token but makes requests already in generation stutter. Much of an inference framework's work is pushing throughput as high as possible while keeping latency within acceptable range. The industry often uses 'Goodput' to describe this goal: the portion of throughput that meets latency requirements.

For agents, there is another calculation to make: the total time of one task is roughly the sum of each step's time-to-first-token, generation time, and tool execution time. A few hundred extra milliseconds per step adds up over dozens of steps to waiting the user can clearly feel. We have viewed the complete trajectory of an agent task in Alibaba Cloud AgentLoop, which displays every model inference and every tool call and marks the time and token consumption of each. As shown below. This data is used to optimize the inference framework in the evaluation stage.

6. MoE: many parameters, but only a small part is used each time

Many current mainstream large models use the MoE (Mixture of Experts) architecture. Inside the model there are many 'expert' sub-networks, and each token is processed by only a few of them. For DeepSeek-V3, the model has 671 billion total parameters, but each token actually activates only about 37 billion parameters. It has a large knowledge capacity with relatively small computation per token. The cost is that all experts must stay resident in GPU memory, and one machine often cannot hold them. Experts must be spread across multiple GPUs. Cross-card communication and load balancing become new problems the inference framework must solve; this is covered in detail in Part 4.

Mainstream Open-Source Frameworks and How They Are Developing

Counting from the explosion of large-model applications in 2023, open-source inference frameworks have developed for only about three years, but several clear threads have formed: general-purpose server-side inference engines, local-run tools for personal computers and edge devices, the orchestration layer that manages entire inference clusters, and inference systems open-sourced by model vendors and Internet companies. This section introduces the origins and development of major projects by these threads, without ranking them. GitHub star counts in the text are as of October 2026 and only reflect community attention.

1. General-purpose inference engines: from academic projects to industry infrastructure

Server-side general-purpose inference engines are currently the most widely used category of inference framework.

Project	Initiator	First public release	Main features	GitHub stars
vLLM	UC Berkeley Sky Computing Lab	June 2023	Proposed PagedAttention for paged KV Cache management; became a PyTorch Foundation-hosted project in May 2025	About 94,000
SGLang	LMSYS	January 2024	Proposed RadixAttention, automatically reuses common prefixes between requests; emphasizes structured generation	About 37,000
TensorRT-LLM	NVIDIA	October 2023	Deeply optimized for NVIDIA GPUs	About 15,000
LMDeploy	Shanghai AI Laboratory InternLM team	2023	Integrates model compression, deployment, and serving	About 8,000

Another project worth mentioning is Hugging Face's TGI (Text Generation Inference), which appeared as early as 2022 and was one of the earliest inference serving frameworks for large models. In December 2025, TGI announced it had entered maintenance mode and recommended users move to inference engines such as vLLM and SGLang.

Looking back at recent years, one obvious feature is how close academic research and engineering practice have become. Continuous batching came from the Orca paper in 2022, paged KV Cache management came from the PagedAttention paper in 2023, FlashAttention for accelerating attention computation was published in 2022, the two foundational papers on speculative decoding appeared from late 2022 to early 2023, and the representative PD-separation studies DistServe and Splitwise were published in 2024. Almost all of these results entered mainstream inference engines very quickly after publication.

Another feature is that projects borrow from each other, and core functions are increasingly converging. Continuous batching, paged KV Cache, prefix caching, chunked prefill, speculative decoding, and structured output are now common configurations in mainstream inference engines. The competitive focus is shifting to other dimensions: how quickly new models are supported after release, how many kinds of hardware are covered, and how well the framework behaves in large-scale cluster deployment.

2. Local and edge deployment: making models run on personal computers

Run models on personal computers and even phones.

llama.cpp was released by developer Georgi Gerganov in March 2023. It is implemented in C/C++ and can run large models on consumer hardware such as ordinary CPUs and Apple silicon. The GGUF format it introduced in August 2023 later became one of the most common file formats for quantized local models. As of October 2026, llama.cpp has more than 130,000 GitHub stars.

Ollama was released in 2023 and packaged model download, running, and a simple API into one command, greatly lowering the barrier to running models locally. It initially wrapped llama.cpp and from 2025 gradually moved to its own engine built on the GGML tensor library. Ollama has more than 180,000 GitHub stars, the highest among inference-related projects.

In addition, MLC-LLM and other projects use machine learning compilation to deploy models to phones, browsers, and more device types.

The star counts of local-run tools are higher than those of server-side inference engines, which to some extent reflects a difference in audiences: individual developers running models on their own computers are far more numerous than engineers who need to deploy inference clusters.

3. Cluster orchestration layer: competition moves from a single machine to a cluster

After 2025, a new category of projects appeared in the inference framework field. They do not replace inference engines but sit above them and manage the whole inference cluster: route requests to suitable instances, implement separate deployment of Prefill and Decode, manage and transfer KV Cache across machines, and autoscale according to load.

At the GTC conference in March 2025, NVIDIA released the open-source distributed inference framework Dynamo, with PD separation, intelligent routing that is aware of KV Cache distribution, KV Cache management and high-speed transfer, and compatibility with inference engines such as TensorRT-LLM, vLLM, and SGLang.

In May 2025, Red Hat, together with CoreWeave, Google Cloud, IBM Research, NVIDIA, and others, initiated the llm-d community to build a distributed inference solution natively running on Kubernetes based on vLLM.

Moonshot AI's Mooncake is the inference serving platform behind Kimi. It uses a KV Cache-centric PD-separated architecture. It published a technical report in June 2024 and has since open-sourced core components such as the data transfer engine and KV Cache storage. Related papers won the best paper award at FAST, a top storage conference, in 2025.

The emergence of these projects is directly related to two changes described earlier: models are getting too large for one machine, and the long context and highly repetitive load brought by agents make cross-request and cross-machine KV Cache reuse more valuable. When single-machine engine optimizations converge, more and more of the inference efficiency gap is opened up by cluster scheduling.

4. Model vendors and Internet companies open-source their inference systems

Model vendors and large Internet companies are open-sourcing the inference technology they use in production.

DeepSeek is a representative example. In February 2025, DeepSeek held an open-source week and open-sourced inference-related components including the attention operator FlashMLA, the expert-parallel communication library DeepEP, the matrix multiplication library DeepGEMM, the expert load-balancing tool EPLB, and the distributed file system 3FS, and also published a design overview of the V3/R1 inference system. In April 2025, DeepSeek explained that its internal inference engine came from an early branch of vLLM from more than a year earlier and was deeply coupled to internal infrastructure, so it chose to contribute independent components and optimizations to the open-source community and to synchronize with the community before new model releases, aiming to make new models deployable on release day. In October 2026, DeepSeek formally open-sourced the inference framework DeepInfer.

In addition to Moonshot AI's Mooncake mentioned above, Alibaba open-sourced RTP-LLM.

These open-source practices take different paths. Some contribute components, others open-source complete systems. Behind them is a shared recognition: the more complex the model structure and the larger the deployment scale, the more the inference-side implementation becomes part of the model's capability.

5. Why new models must wait for inference framework 'adaptation'

Many engineers have had this experience: a new model is released, the weights are downloaded, but the commonly used inference framework does not support it yet, or it runs but gives wrong results or runs very slowly. This is because an inference framework cannot automatically run any model; every new model requires specific adaptation work.

There is a lot to do. First, implement the model structure. New models often introduce new attention mechanisms and new expert routing methods, so the inference framework must implement the corresponding computation logic and write efficient kernels; otherwise, even if it runs, it will be slow. Second, align the chat template and tool-call format. As described earlier, a wrong template reduces output quality and can make tool calls fail frequently. There are also new quantization formats, new multimodal input methods, and more.

Two recent examples illustrate the point. In September 2025, DeepSeek released the V3.2-Exp model with sparse attention, and vLLM announced support on the day of release. In the same month, Alibaba's Tongyi Qianwen released Qwen3-Next with a hybrid attention structure, and vLLM announced support the next day. This speed is possible because model vendors cooperate with inference framework communities before release. This kind of 'deployable on release day' collaboration is becoming an industry norm.

For engineers, this means that when deploying a new model, pay special attention to the inference framework version. In the early period after a new model release, support may differ across frameworks and versions, and performance and output quality may still be under optimization. Models and inference frameworks are increasingly designed and released together.

6. Hardware diversification: from NVIDIA GPUs to domestic chips

For a long time, large-model inference ran mainly on NVIDIA GPUs, and most inference framework kernels and optimizations were built around NVIDIA's CUDA ecosystem. This is changing.

On one hand, inference frameworks are leaving architectural room for multiple hardware backends. vLLM, for example, introduced a hardware plugin mechanism in 2025, allowing chip vendors to maintain their own hardware backends outside the main project. Based on this mechanism, the community maintains vllm-ascend for Huawei Ascend, Baidu released vLLM-Kunlun for Kunlun chips, and MetaX has also released an adaptation. The JD xLLM mentioned earlier focuses on supporting domestic chips, and DeepInfer supports both NVIDIA GPUs and Huawei Ascend NPUs.

On the other hand, model vendors are also beginning to design models with the characteristics of domestic chips in mind. In August 2025, when DeepSeek released V3.1, it said the UE8M0 FP8 numeric format used by the model was designed for the next generation of domestic chips.

In addition, higher-level operator programming languages such as Triton and TileLang let developers avoid writing low-level code for each chip by hand, which to some extent reduces the cost of porting inference frameworks to new hardware.

For enterprises that need to deploy models on domestic chips, these developments mean more available options, but support scope and performance still vary considerably across different chip and framework combinations, and each one must be verified before deployment. The more diverse the hardware, the more valuable the inference framework becomes as the adaptation layer between models and chips.

7. Summary: three directions of change

Looking at the development of open-source inference frameworks in recent years, three directions stand out: from single-machine engines to cluster systems, from generic implementations to co-design with models, and from a single hardware platform to multiple hardware platforms. Behind all three is the same driving force: large models are moving from a technology tried by a few enthusiasts to large-scale, continuously running infrastructure, and inference efficiency has become the key to whether this can be sustained.

Inference Framework Architecture: How a Request Is Processed

The previous sections introduced the concepts involved in inference frameworks. This section changes the angle: follow one request from sending to return, see what happens inside the inference framework, and what problem each stop solves.

Assume you are using an enterprise internal customer-service agent. You enter a question. The agent packs the system prompt, definitions of a dozen tools, records from previous conversation turns, and your new question into a request of about 8,000 tokens and sends it to the model service.

1. Entry: from API protocol to token sequence

The request first arrives at the access layer of the inference framework. Most model APIs today are compatible with several mainstream protocol formats. The access layer's first job is to convert requests in different protocols into a form the model can understand.

The core of this step is the chat template. The system prompt, user messages, assistant replies, tool calls, and tool return results in a request must be concatenated into a complete text sequence using a series of special markers according to the format agreed upon during model training, and then converted into token IDs by the tokenizer. Every model has a different chat template. If the template is wrong, the model may still output text, but quality can drop noticeably and tool calls may fail frequently.

The access layer also carries a security responsibility that is easy to overlook. The special markers in chat templates act as 'control instructions' to distinguish who is speaking and where a tool call begins and ends. If user input or tool return content happens to contain a string identical to a special marker, the access layer must treat it as ordinary text; otherwise it could be exploited to fake roles or inject instructions.

When the response is returned, the access layer does the reverse work: converts output tokens back into text, extracts the reasoning process, body text, and tool-call arguments, and returns them incrementally in streaming format according to the corresponding protocol.

This layer looks like pure format conversion but is the source of many production problems. The access layer determines whether the same model deployed in different places behaves consistently.

2. Scheduling: putting requests into batches

After conversion into token sequences, the request enters the scheduler's waiting queue. The scheduler is the 'command center' of the inference framework. Before each computation step, it makes a series of decisions: which requests should be processed in this step? Should a newly arrived request start Prefill now? Should requests already in generation continue? Is there enough KV Cache space in GPU memory?

Continuous Batching is implemented here. The scheduler works at the granularity of each generation step, moving finished requests out of the batch and filling waiting requests in, keeping the GPU fully loaded.

Our 8,000-token request puts the scheduler in a dilemma. If it Prefills the whole request at once, other requests currently generating in the batch must stop and wait, and users will feel obvious stutter. A common approach is Chunked Prefill: cut the long request into several smaller chunks and process only one chunk per step, mixing it with Decode of other requests. This both keeps new requests from waiting too long and keeps old requests from stuttering.

The scheduler must also consider memory capacity. If KV Cache space is insufficient, new requests can only keep queuing. In extreme cases, the scheduler may even pause some requests that are generating, swap their KV Cache out to host memory, or drop it entirely and recompute later when space is available.

Scheduling itself also has a cost. Scheduling decisions, preparing input data, and processing generated results require CPU work. If the GPU has to wait for the CPU to finish before starting each next step, it idles. Current inference frameworks generally use asynchronous scheduling: the CPU prepares the next step while the GPU computes the current one, squeezing out the GPU's idle gaps.

In multi-machine deployment, scheduling has two levels. The cluster-level load balancer first decides which inference instance should receive the request, and the instance-level scheduler decides how and when the request is computed. The load balancer must consider each instance's queue length, memory usage, and whether a prefix of the request is already cached on an instance. Taking Alibaba Cloud PAI-EAS's LLM intelligent routing as an example, it provides multiple routing policies such as by prefix cache, by fewest requests, and by fewest tokens. The prefix-cache policy sends requests with the same context preferentially to the instance that already caches the corresponding KV Cache. The scheduler must always answer the same question: what should the GPU compute next.

3. Execution: models, parallelism, and kernels

After the scheduler determines the batch, the request enters the execution layer and the actual model computation begins.

When a model cannot fit on one GPU, the execution layer must first split it across multiple GPUs. This is parallelism. There are four common approaches: Tensor Parallelism (TP) splits each layer's large matrix across multiple GPUs for joint computation; Pipeline Parallelism (PP) assigns different layers to different GPUs, working like an assembly line; Data Parallelism (DP) lets multiple GPUs each process different requests with a full model; Expert Parallelism (EP) is for MoE models and distributes different experts across different GPUs. For moderate model sizes, tensor parallelism on one or a few GPUs is enough. For large MoE models, data parallelism and expert parallelism are often combined, reaching scales of dozens or hundreds of GPUs. Tokens must be sent to experts on different GPUs for computation, requiring frequent data exchange between cards. Some experts are especially 'hot', and the cards holding them slow down the whole batch; inference frameworks usually replicate hot experts several times so every card stays as busy as possible.

Every layer of model computation eventually lands on kernels: low-level computation programs executed directly on the GPU, such as matrix multiplication, attention, expert routing, and communication. For the same computation task, the quality of kernels can differ by several times in performance. There are many classic kernel optimizations for inference, such as FlashAttention, which reduces GPU memory reads and writes to speed up attention. Model vendors and hardware vendors also write specially optimized kernel libraries for particular model structures and chips. Low-precision computation also lands at this layer: matrix multiplication at 8-bit or even 4-bit precision saves memory and can use the low-precision compute units of newer GPUs for speed.

The execution layer also has invisible optimizations. Every GPU kernel launch has a fixed overhead, and in Decode each step's computation is small, so kernel launch overhead accounts for a large share. Inference frameworks 'record' a whole step's kernel calls and replay them at once to reduce launch overhead. The model structure decides what must be computed; kernels decide how fast it is computed.

4. Caching: do not compute again what has already been computed

In our 8,000-token request, the system prompt and tool definitions are the same every time, and the earlier conversation turns were already computed in the previous request. If the KV Cache for this content is still saved, it can be reused directly, and only the small new part needs to be computed. This is Prefix Caching, the core value of the caching layer.

To reuse KV Cache efficiently, it must be managed well. Early inference frameworks reserved one contiguous block of GPU memory per request to store KV Cache. Because the eventual generation length was unknown, memory was reserved at maximum length, wasting a lot of memory. Later, paged management borrowed the idea of OS memory management, cut KV Cache into fixed-size 'pages', allocated them on demand, and did not require pages to be contiguous. This greatly reduced memory waste, let one GPU serve more requests at the same time, and made it easy for different requests to share KV Cache for the same prefix.

GPU memory is limited, and the amount of KV Cache that can be saved is ultimately small. So inference frameworks began to build multi-level caches: the most frequently used content in GPU memory, the less frequently used in host memory, and more on SSD or even in a separate distributed storage cluster. At the cluster level, load balancers try to route requests to instances that already cache their prefix, or all instances share one cache pool. Alibaba Cloud's Tair KVCache belongs to this kind of cache pool: it pools the GPU memory and host memory of GPU servers, extends KV Cache from GPU-memory-only to tiered caching across GPU memory, host memory, and storage, and adapts to inference engines such as vLLM, SGLang, and TensorRT-LLM.

Understanding this lets you read the 'cache hit' price on model API bills. Input tokens that hit cache skip Prefill computation, the provider's cost drops significantly, so many vendors give large discounts for cache-hit input tokens, commonly 10% to 20% of the original price.

For agent developers, there is a very practical lesson: keep the beginning of every request stable, put fixed system prompts and tool definitions at the front, and append new content at the end. If you insert something that changes every time, such as the current time, at the beginning of the system prompt, the cache for everything after it is invalidated. In agent scenarios, cache hit rate is cost.

5. Cluster: from one machine to a cluster

When a model is too large for one machine and traffic is too high for one instance, the inference framework turns from a program into a distributed system.

The first change is separate deployment of Prefill and Decode, or PD separation. As described earlier, the two phases are completely different: Prefill is compute-hungry and Decode is bandwidth-hungry. On the same group of GPUs they interfere with each other; the Prefill of a long request can slow down all currently generating requests. PD separation uses one group of machines dedicated to Prefill and another group dedicated to Decode. After Prefill completes, the generated KV Cache is sent to the Decode machines over a high-speed network to continue generation. The two groups can each choose the most suitable parallelism and scale, optimizing for time to first token and generation speed separately. In the PAI-EAS PD-separation template for MoE models, Prefill instances default to 8-card tensor parallelism and Decode instances default to 8-way expert parallelism plus 8-way data parallelism, and users can adjust as needed. The platform can also recommend the Prefill-to-Decode instance ratio and parallelism strategy based on model characteristics, machine type, and input/output length. For multimodal models, image and video encoding can also be split out and deployed separately.

The second change is larger-scale parallelism and more complex load balancing. For example, the Decode phase of a large MoE model may need dozens or hundreds of GPUs as one instance. Whether experts are balanced and whether inter-GPU communication is efficient directly decides overall throughput.

The third change is elasticity and fault tolerance. Traffic has peaks and valleys. An inference cluster must scale in and out according to queue length, while keeping some redundancy for burst traffic. At large scale, hardware failure is the norm. In an instance composed of hundreds of GPUs, any single GPU failure can make the whole instance unavailable. The inference framework must be able to quickly move affected requests to other instances and restore service with the remaining healthy hardware as soon as possible. Single-machine inference competes on computing fast; cluster inference competes on scheduling well.

6. Sampling and output: making results controllable and reproducible

After each computation step, the model does not output a fixed character. It outputs a probability distribution over every token in the vocabulary. The process of selecting the next token from that distribution is called sampling. This is the last stop in the request's path through the inference framework, and it decides whether the output is controllable.

Several parameters control the randomness of sampling. A higher temperature makes selection more random and output more diverse; a lower temperature favors the highest-probability token and makes output more stable. Top-p and Top-k limit selection to only the highest-probability portion of tokens.

This explains a common confusion: why does asking the same question twice produce different answers? First, sampling itself is random. But even with temperature set to 0, many inference services still produce slightly different results. The reason is that floating-point operations on GPUs are sensitive to computation order. When the same request is computed in a batch with different requests, the order of operations inside a kernel may differ, producing extremely small numerical differences. Occasionally these differences cause the model to choose a different token at one step, and everything after changes. For this reason, some inference frameworks are beginning to offer 'batch-invariant' or deterministic inference modes, guaranteeing the same output for the same input no matter which batch it is placed in, at the cost of some performance. This is important for reproducibility of evaluation results and also for reinforcement learning training, because training needs inference and training phases to produce consistent probabilities for the same content.

Sampling can also do two valuable things.

One is structured output. When an agent calls a tool, the model must output JSON that strictly meets format requirements; even one missing bracket makes the program unable to parse it. Structured output compiles the format requirement into a grammar. Before each sampling step, it computes which tokens are legal based on the content generated so far and directly excludes illegal tokens, fundamentally guaranteeing correct output format.

The other is speculative decoding. The Decode bottleneck is 'flipping the book', and the compute units have a lot of idle capacity. Speculative decoding first uses a small, fast draft model (or a lightweight prediction module attached to the main model) to guess the next several tokens at once, then lets the large model verify all guesses in parallel. Guesses that are correct are kept directly; if one is wrong, generation restarts from the error position. Mathematically, this method guarantees that the final output probability distribution is exactly the same as the large model generating token by token, while increasing generation speed. The more accurate the guesses, the more obvious the speedup.

Finally, generated tokens are sent back to the access layer, converted to text, parsed into the format the protocol specifies, and returned incrementally to the agent in streaming mode. At the same time, the inference service records the request's input token count, the part that hit cache, and the output token count for billing and monitoring. Sampling is the last stop of a request; it determines whether the model's output can be reliably used by programs.

7. Summary: the journey of a request

Stage	Main work	Problem solved	Main impact
Entry	Protocol conversion, chat template, tokenization, output parsing	Make the request enter the model in the correct format	Output consistency and security
Scheduling	Continuous batching, chunked prefill, memory admission, load balancing	Decide what the GPU computes next	Throughput, time to first token, stutter
Execution	Parallelism, kernels, low-precision computation	How fast each step is computed	Single-GPU performance, hardware utilization
Caching	Paged management, prefix reuse, multi-level cache	Do not compute again what was computed	Cost and time to first token
Cluster	PD separation, large-scale parallelism, scaling and fault tolerance	Remain stable and efficient after scaling	Overall throughput and stability
Sampling and output	Sampling strategy, deterministic inference, structured output, speculative decoding	Whether output is controllable, reproducible, and fast enough	Reliability and generation speed

Who Uses Inference Frameworks

1. Model vendors and cloud vendors: inference efficiency is competitiveness

For model vendors and cloud providers that offer model APIs, the inference framework is core infrastructure. On the same group of GPUs, if inference efficiency doubles, it can serve twice as many users, or cut the price in half. Price competition in model APIs is to a large extent competition in inference efficiency. Such companies usually have dedicated inference optimization teams that deeply customize open-source frameworks or build their own, with optimizations specific to their own model structures.

2. Enterprises that need private deployment: data does not leave the domain

Enterprises in finance, government, healthcare, energy, manufacturing, and other industries often cannot send data to external model APIs due to data security and compliance requirements. They need to deploy open-source models in their own data centers or private clouds. These enterprises are direct users of inference frameworks: they must select, deploy, tune, and operate them themselves, and increasingly they must do so on domestic chips. Many choose software-hardware integrated inference appliances or private inference platforms and hand the complexity of inference frameworks to suppliers. But understanding the basic principles of inference frameworks still helps them make more reasonable hardware selection and capacity planning decisions.

3. Teams building agents and applications: indirect users, direct beneficiaries

Most agent and AI application teams call models through APIs and do not directly deploy inference frameworks. But inference framework characteristics genuinely affect their costs and experience. When choosing a model service provider, time to first token, generation speed, and cache-hit price are important considerations. When designing prompts and context structure, whether the layout is friendly to cache hits can make costs differ by several times. When structured output is needed, whether structured output is supported determines the reliability of tool calls. To figure out how much each factor matters in their own agent, teams can use observability tools. For example, Alibaba Cloud AgentLoop records each step of a model call, tool call, and retrieval in an agent, tracks the token consumption and time of each model inference, and helps locate which step is slow and where money is spent. In addition, more teams are deploying small models themselves for some agent steps, such as intent classification and information extraction, to reduce cost and latency, and they must then deal with inference frameworks directly.

4. Post-training and research teams: inference inside the training pipeline

Teams doing model post-training are another heavy user category of inference frameworks. Reinforcement learning requires the model to generate many trajectories, distillation requires a teacher model to generate demonstrations in batches or score a student's output, evaluation requires the model to complete many test questions, and synthetic data requires the model to produce samples in batches. These scenarios care more about offline batch processing throughput and also have higher requirements for consistency between inference results and the training process.

5. Individual developers: running models on their own computers

As local-run tools mature and small models become more capable, more individual developers run models on laptops or desktop computers for learning, development and debugging, or for processing private data that is inconvenient to upload to the cloud. Edge inference is also extending to phones, personal computers, cars, and other devices. These scenarios require inference frameworks that are easy to install, use few resources, and can run on consumer hardware.

Conclusion

Back to the opening question: what is the relationship between inference frameworks, models, and agents?

An inference framework is inside the model training pipeline, responsible for sampling in reinforcement learning, evaluation, and data synthesis. It is also after the model launches, taking every user question and agent call. It connects agent applications above with the compute cluster below, delivering the model's capability to every request at the lowest possible cost and the highest possible speed.

The more agents become widespread, the more often models are called, and the longer the context of each call. In this multiplication problem, every improvement in inference efficiency is amplified by call volume.