Frontend

DeepInfer's Open Source Code: A One-Model Engine Built for 155T Tokens a Day

By moshuying · Oct 22, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

DeepInfer ships without an OpenAI-compatible endpoint; the repo offers offline batch generation only, with cross-instance KV cache, request routing, load balancers, and ops tooling absent. On SM100 or Ascend 950, it still works as a reference implementation for how a production MoE engine schedules prefill and decode, prices speculative verification, and balances expert load.

Summary

DeepInfer is DeepSeek's open-source inference stack for its V4.1 Flash model, and the repository is deliberately narrow. The model registry contains one entry, DeepseekV41ForCausalLM, the default checkpoint is DeepSeek-V4.1-Flash-1017, and the C++ scheduler, block manager, and model modules are tied to that architecture. The Python API still carries vLLM names like LLM, EngineArgs, Sequence, and SamplingParams, but the engine underneath is a custom rewrite.

The source code rewards careful reading. The step() loop at llm_engine.py:860 invokes a scheduler that uses ALL/ANY voting across DP ranks to avoid allreduce deadlocks when prefill and decode share one process. GPU memory is managed in fixed-size blocks with reference counting, prefix sharing, and block-level preemption. DSpark adds a confidence head to speculative decoding and uses runtime estimators to decide how many draft tokens to verify; official numbers put the gain at 4 to 7% throughput. EPLB replicates hot experts and targets the historically busiest rank instead of average load, so MoE tail latency stays in view.

What is not in the repository matters as much as what is. The architecture diagram shows a protocol layer, three load balancers, and 3FS-based cross-instance KV storage; none of those ship. The open-source code exposes offline LLM.generate() batch inference, one model, NVIDIA SM100 or Ascend 950 support, and no OpenAI-compatible HTTP service. The 155-trillion-token headline belongs to the internal deployment, not to a single checkout of this repo.

Takeaways

— DeepInfer's model registry accepts one model, DeepseekV41ForCausalLM, and the default checkpoint is DeepSeek-V4.1-Flash-1017.

— Contact with the outside world is offline generation through LLM.generate(). No OpenAI-compatible HTTP server, prefix router, cross-instance KV cache, autoscaling, auth, or rate limiting exists in the repository.

— Hot-path scheduling lives in a rewritten C++ layer at csrc/engine/scheduler.hpp:275; the Python API keeps vLLM names such as LLM, EngineArgs, Sequence, and SamplingParams.

— In NORMAL mode, scheduler voting uses ALL and ANY checkpoints so every DP rank picks the same prefill or decode action; otherwise collective allreduce operations can deadlock.

— GPU memory uses fixed-size blocks in block_manager.hpp with allocate, free, and can_allocate_nolock; per-sequence block count is (prompt length + max generated tokens + block size - 1) / block size + 1.

— DSpark's confidence head predicts acceptance for each draft token, then runtime estimators pick the verification length. Official results: verification length drops from a fixed 5 to 3.2-4.4, accepted length stays about even at 3.29 vs 3.20, and throughput improves 4-7% around 60 concurrent requests per rank.

— EPLB replicates hot experts and greedily places the replicas; the load-balance metric (slowest rank divided by average) reaches 1.074 for prefill at EP4DP4, 1.045 for prefill at EP2DP2, and 1.090 for decode.

— YOCO's DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140 is a bounded suffix replay for the consumer path, not a context truncation; with the default nextn=5 the lower bound is 134.

— CUDA and Ascend get mirrored operator sets: 30 TileLang ops in cuda/ and 30 matching ops in asc/, plus vendor submodules such as DeepGEMM-Ascend and DeepEP-Ascend.

— Deterministic sampling needs DEEPINFER_BATCH_INVARIANT=1, DEEPINFER_ENABLE_MTP_VARLEN=0, and a fixed seq_seed. Batch-invariant mode only promises identical results along one fixed trajectory, not identical arbitrary random runs.

— Official prefill throughput stops climbing around 32,768 tokens per DP rank; decode per-GPU throughput goes from 2,500 to 26,000 tokens/s as per-rank concurrency rises 8 to 256, while per-user speed drops from 325 to 105 tokens/s.

— Builds require Linux, Python 3.12, GCC 13+, CUDA 13 or CANN 9.2, and build.sh force-installs torch==2.13.0+cu132 and NCCL 2.32.3, which can corrupt an existing vLLM environment.

Conclusions

DeepSeek released the engine but kept the distributed control plane behind the headline number; the stubbed methods init_pd_split(), do_save_kvcache(), and send_processed_prompts() are explicit extension points for a system that did not ship.

The ALL/ANY voting scheme is a code-level consequence of collapsing separate prefill and decode services into one offline process; it shows production scheduling decisions are shaped as much by collective-communication constraints as by throughput.

DSpark's 4-7% throughput gain is small enough to suggest speculative decoding is reaching diminishing returns on an already tuned model, but the transferable idea is pricing verification work: predict acceptance, estimate runtime, and choose batch size from expected output per unit time.

EPLB's choice to optimize the historically busiest rank rather than average load treats tail latency as the binding constraint; the same principle would change how replicas are placed in any MoE cluster.

The official charts are best-case steady-state measurements with controlled routing, selected measurement windows, and no same-config comparison against vLLM or SGLang, so they establish scalability, not superiority.

Concepts & terms

YOCO (You-Only-Cache-Once)

A V4.1 layer scheme where Encoder layers compute KV for prefill and Consumer layers reuse previously cached state, avoiding repeated recomputation. Around cache hits it needs bounded replay of a short suffix rather than full recomputation.

EPLB (Expert Parallel Load Balancer)

Balances MoE expert load by adding redundant replicas of hot experts and greedily placing physical replicas across ranks; its objective targets the historically busiest rank to reduce tail latency.

DSpark

DeepSeek's speculative decoding variant that adds a Confidence Head to predict acceptance rates for draft tokens, then uses runtime estimators to choose how many draft tokens the large model should verify.

Chunked prefill

Splitting a very long prompt into chunks so a prefill step processes a bounded number of tokens; DeepInfer's official chart shows throughput plateauing around 32,768 tokens per DP rank while latency keeps climbing.

EPD (Expert-Parallel Disaggregation)

Separating prefill, decode, and vision-encoding stages, each with its own expert-parallel placement; the architecture diagram includes it but the open-source repository does not ship all three stages.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗