DeepInfer's Open Source Code: A One-Model Engine Built for 155T Tokens a Day

DeepSeek open-sources production-grade inference framework DeepInfer: what does one cluster processing 155 trillion tokens a day mean?

DeepSeek has open-sourced the inference framework that runs its online service. It is called DeepInfer.

The official claim: a single cluster handles 155 trillion tokens per day, and at off-peak electricity prices each card earns $38 per hour. My first reaction to those numbers was to go read the code and see whether they add up.

After reading through it, my feelings are split. On the one hand, the scheduler and speculative decoding are genuinely well written and worth reading line by line. On the other hand, the pretty architecture diagram has three large blocks that do not exist in this repository.

This post follows the order in which I read it. Where something matches the source code I give line numbers, and where it does not, I say so.

It only runs one model

DeepInfer is not a general-purpose serving framework in the style of vLLM. The model registry contains exactly one entry, DeepseekV41ForCausalLM, and the default checkpoint is DeepSeek-V4.1-Flash-1017.

	DeepInfer	vLLM / SGLang
Supported models	One	Hundreds
Entry form	Offline LLM.generate()	OpenAI-compatible HTTP API
Hardware	NVIDIA SM100 / Huawei Ascend 950	Broad GPU models
Optimization approach	Tied to V4.1 structure	Generic abstractions plus plugins

The README itself says it was originally built based on vLLM, and also referenced SGLang and TensorRT-LLM. That is true, but only half the story.

The Python layer still has vLLM's shape: LLM, EngineArgs, Sequence, SamplingParams, and even the comments look similar. But the scheduler, block management, and input assembly under csrc/engine/ are rewritten C++, and the model structures in model_executor/models/v4_modules/ are written specifically for CED, YOCO, Engram, and DFlash.

A vLLM shell housing an engine customized for DeepSeek's own model. I think that analogy is fairly accurate.

The architecture diagram does not picture this repository

Look at the green block first: DeepSeek Recipe. It is the protocol layer. It normalizes the JSON of three APIs, Chat Completions, Responses, and Messages, into dialogs and token sequences and feeds them into the engine; the token stream produced by the engine is also parsed back by it into each API's SSE response. That component lives in another repository, deepseek-recipe, not in DeepInfer.

Red is the Entrypoint, the request entrance. It looks up the request against the KV cache prefix, but only metadata, through a hash chain, and then decides whether the request should go to prefill or decode.

The yellow and orange boxes are the Prefill Service and Decode Service. The former takes uncached prompts and computes KV; the latter emits tokens one by one. Inside each machine there is also an Expert-Parallel Load Balancer that decides how experts are distributed.

Purple is the vision encoder. Images first go through a ViT to produce embeddings, then enter prefill; multimodal runs through EPD's three-stage separation.

The three blue rounded boxes are three load balancers, each routing on a different dimension. The VE one picks the GPU with the fewest image patches. The Prefill one splits by context length into four tiers (<16K, 16-64K, 64-256K, >=256K) and dynamically allocates according to the previous round's compute quota. The Decode one looks at both request count and total KV length, sending short requests to cards with low concurrency but long KV.

At the bottom in gray is the KV storage cluster on 3FS. The thick blue arrows are cache reads and writes: prefill writes to disk when done, decode pulls as needed.

The whole diagram is logically coherent. The problem is that the red routing box, the three blue boxes, and the storage layer underneath are not in this repository.

In LLMEngine you can find methods such as init_pd_split(), do_save_kvcache(), and send_processed_prompts(). Click into them and each one is either pass or a default implementation. These are extension points reserved for the internal system, not open-source features.

The repository has five layers

Looking at the directories:

Directory	What it does	Size
deepinfer/engine/	Engine main loop, parameter parsing, statistics logging	6 py
deepinfer/worker/	Worker, ModelRunner, CUDA Graph, CacheEngine	10 py
deepinfer/model_executor/	Model structure, operators, parallel communication, EPLB	24 + 33 + 30 + 30 py
deepinfer/ve/	Multimodal preprocessing	9 py
deepinfer/structure_output/	Compilation and matching for structured output	5 py
csrc/engine/	Scheduler, block management, input preparation, sampling preparation	About 20 cpp/hpp
csrc/graph_timer/	CUDA Graph / Ascend profiler timing	4 files
deep_cluster/	Hardware probing, NUMA binding	5 py
vendor/	6 submodules: FlashMLA, DeepGEMM, DeepEP, DeepSelect plus Ascend versions	-

Most of the hot path is in C++. The reason is simple: the scheduler runs once for every token generated, and the decode stage does anywhere from tens to hundreds of steps per second, so Python overhead gets amplified until it is unbearable. scheduler.hpp is over 1,300 lines, all of it on the mandatory path for every step.

How a request flows through

The entry point is LLM.generate() in deepinfer/entrypoints/llm.py. Encode the prompt, call add_request for each item, and loop over step(). That is all three things.

The main loop is step() at llm_engine.py:860. One call produces a batch of tokens:

# llm_engine.py:881
with cpu_timer.record('schedule'):
    seq_metadata_list, scheduler_outputs = self.scheduler.schedule(
        self.sequence_metadata_pool
    )
# ... process clone_states / so state / multimodal fetch ...
request_outputs = self._execute_model_async_step(seq_metadata_list, scheduler_outputs)

Scheduling happens at csrc/engine/scheduler.hpp:275. It has to answer: for this step, prefill or decode, which sequences, and how long a chunk for each sequence.

Execution happens at deepinfer/worker/worker.py:89. It turns scheduling results into GPU tensors, runs forward, samples, and in async mode pushes post-processing to the next round.

A few things in the source I want to remember

The scheduler votes

EngineType has three values: PROMPT does only prefill, COMPLETION does only decode, and NORMAL does both. Offline inference uses the third, which also lines up with the Prefill Service and Decode Service in the architecture diagram.

NORMAL has a very practical problem. All ranks in a DP group must do the same thing; if rank 0 is prefilling while rank 1 is decoding, the allreduce deadlocks.

The solution in the source is voting, at scheduler.hpp:295:

bool is_any_point = (decode_step != 0) && (decode_step % prefill_any_attempt_interval == 0);
bool is_all_point = (decode_step % prefill_attempt_interval == 0);
if (!is_any_point && !is_all_point) {
    return _schedule_decode(acquire);
}

bool want = (get_gpu_cache_usage() <= GPU_CACHE_PREFILL_WATERMARK) && _has_prefill_waiting();
bool do_prefill;
if (is_any_point) {
    // ANY: any rank wants it, trigger
    do_prefill = (_dp_allreduce(want ? 0 : 1, c10d::ReduceOp::MIN) == 0);
} else {
    // ALL: everyone wants it to trigger
    do_prefill = (_dp_allreduce(want ? 1 : 0, c10d::ReduceOp::MIN) == 1);
}

An ALL point appears every prefill_attempt_interval decode steps and requires every rank to have prefill work, so one rank does not work while the others spectate. ANY points occur more frequently; if any one rank has work, everyone does it together, as a safety net for an isolated rank.

There is a second round of voting afterward. want only checks that the waiting queue is non-empty; _schedule_prefill can still come back empty-handed because slots are full or KV blocks are lacking. If the whole group comes back empty-handed it has to switch to decode; otherwise it falls into a loop of doing empty prompt steps that reset decode_step to zero, voting again, and coming back empty-handed again.

Also, GPU_CACHE_PREFILL_WATERMARK is 0.9; once cache usage is past 90 percent it stops accepting prefill.

I have not seen this voting logic in any other analysis post, but it is the most honest compromise when a disaggregated deployment architecture is reduced to the single-machine offline scenario.

GPU memory is managed in blocks

csrc/engine/block_manager.hpp follows roughly the same idea as an OS managing memory pages. GPU memory is split into fixed-size blocks, allocated on demand, reference-counted, and recycled when done.

There are three core methods:

int allocate();           // take one from the free_blocks list
void free(int block_id);  // put it back
AllocStatus can_allocate_nolock(int to_allocate_blocks);

The benefit is direct. Requests vary in length, so there is no need to reserve contiguous memory; identical prefixes can share physical blocks; when a request is preempted, only its blocks are freed, no data is moved.

The number of blocks a sequence needs is calculated at scheduler.hpp:191:

return (seq.get_len() + seq.generation.stopping.max_tokens + block_size - 1) / block_size + 1;

Prompt length plus maximum generated length, rounded up, plus one spare block.

Async scheduling saves waiting, not compute

The official name is the zero-overhead batch scheduler. After reading the code, I think that claim needs a discount.

The original pipeline was:

schedule(N) -> forward(N) -> sample(N) -> [wait for GPU result] -> post-process(N) -> schedule(N+1)
                                          ^ GPU idle here

With async execution, while the GPU runs forward(N) and sample(N), the CPU concurrently runs schedule(N+1), launches the next round of forward, and handles the previous round's post-processing. The waiting gap gets filled.

But CPU overhead does not disappear. Event synchronization, metadata processing, and resource reclamation are all still there; it just changes from GPU waiting on CPU to CPU hiding inside GPU execution time.

A third-party analysis put it more bluntly: zero overhead is the design goal, not the fact. I agree.

The corresponding switch is DEEPINFER_ZERO_OVERHEAD_SCHEDULING.

DSpark sells verification compute

Ordinary speculative decoding is crude: the draft model emits 5 tokens at once, the large model verifies them in parallel, and however many are accepted, they are accepted.

But each draft token's acceptance probability differs a lot. Fixed verification of 5 is like spreading compute evenly across a bunch of low-probability tokens.

DSpark adds a Confidence Head that predicts the acceptance rate for each draft position, then decides how many to verify. The entry is at csrc/engine/mtp_varlen.hpp:150:

void schedule(const std::vector<std::shared_ptr<Sequence>> &scheduled,
              std::vector<int> &seqlens_q) const {
    ...
    // maintain two queues, one for computing best_bsz, one for computing seqlen_q
    std::vector<std::pair<float, int>> acc_queue_for_bsz;
    std::vector<std::pair<float, int>> acc_queue_for_seqs_q;

It queues each draft position by confidence, enumerates verification sizes, and picks the one with the largest expected output per unit time.

How is output estimated? Two runtime estimators. ForwardRuntimeEstimator (mtp_varlen.hpp:48) uses recently observed forward latencies to correct the curve profiled at startup. SamplerRuntimeEstimator (mtp_varlen.hpp:20) uses a sliding-window median, preferring samples under the same request count and falling back to global when samples are insufficient.

There is an easy-to-get-wrong detail, spelled out clearly in the comment:

// Need to ensure confidence_{i+1, ..., nextn} does not affect whether draft_token_i is verified.
// Suppose we are in round N; we expect to use round N-2's next_n_token_accept_rate to compute best_bsz,
// and round N-1's next_n_token_accept_rate to allocate each request's seqlen_q

Two constraints stack. Spatially, later positions' confidence must not affect whether earlier positions are verified; otherwise the logic contradicts itself. Temporally, in the async pipeline, when round N is scheduled, round N-1's results are not yet back, so batch size is estimated with round N-2's acceptance rate and per-request verification length is estimated with round N-1's.

To avoid missing accounting, each request carries two counters, schedule_cnt and process_cnt, plus an assertion:

TORCH_CHECK(mtp.process_cnt <= mtp.schedule_cnt && mtp.schedule_cnt <= mtp.process_cnt + 1, ...);

A difference of 0 means the pair is matched and round N-1 data is used; a difference of 1 means the latest forward has not yet produced an acceptance rate, so it has to roll back one more round.

Official data: verification length drops from a fixed 5 to 3.2-4.4, average accepted length is basically unchanged (3.29 vs 3.20), the production operating point is about 60 concurrent requests per rank, and throughput improves 4% to 7%.

Set DEEPINFER_ENABLE_MTP_VARLEN=1 to turn it on; maximum draft length is controlled by DEEPINFER_USE_NEXTN_DECODE and defaults to 5.

EPLB watches the busiest card

MoE's problem is uneven expert popularity. Some experts get routed to constantly, others are mostly idle, and in the end every card waits for the slowest one.

EPLB's approach is to add redundant replicas for hot experts. eplb/policy.py works in two steps: first decide which experts to replicate based on token load over the recent N steps, then greedily place physical replicas onto ranks.

There is a detail in the second step's objective function that I think matters: it tracks the historically busiest rank, not the long-run average load. Averaging misses tail latency, and tail latency is what stalls the whole batch.

The number of redundant replicas has to be divisible by ep_size:

def resolve_num_extra_experts(num_routed_experts, min_num_extra_experts, ep_size):
    """Return the number of extra experts, not below the lower bound, that makes num_physical_experts divisible by ep_size."""
    padding = -(num_routed_experts + min_num_extra_experts) % ep_size
    return min_num_extra_experts + padding

The reordering is done by expert_model_parallel_shuffle_inplace, which moves weights directly via NVLink symmetric memory, not over the network.

Official results: prefill moe_lb 1.074 (EP4DP4) and 1.045 (EP2DP2), decode 1.090. This metric is slowest rank divided by average; 1.0 is perfect.

YOCO's 140 is not a truncation

V4.1 layers come in two kinds: Encoder layers that actually do prefill computation, and Consumer layers that reuse previous state (YOCO, You-Only-Cache-Once).

The trouble appears on KV cache hits. If only the block KV is hit, the SWA (sliding-window attention) KV is missing. The naive approach is to recompute the whole segment, but under YOCO, rebuilding decoder state means running the last 2,560 tokens through 20 layers, which is too expensive.

What actually happens is recomputing only the last 128 tokens, with the attention window truncated inside the replay segment. The official wording is:

replaying from position s, the query at position i is computed only against keys in the range max(s, i-127) to i

The sliding window only looks at the latest 127 positions anyway, so replaying earlier data is meaningless. This makes the replay cost constant, independent of context length and number of layers.

The README gives this configuration:

DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact
DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140

I have seen several misreadings of this 140, most commonly that the context is cut to 140 tokens. That is not it at all. It is the suffix replay length for a specific decoder path; it does not truncate user input and does not shrink the model cache.

The constraint is keep_tokens >= window_size + max_backtracking_tokens. Speculative decoding needs to roll back, and the rollback length must keep enough window history. With the default nextn=5, the backtracking margin is 6, so the lower bound is 134; the example uses 140 to leave a few extra tokens.

One logic, two operator sets

Supporting Ascend is not as simple as it runs; both kernel sets are optimized.

deepinfer/model_executor/layers/tilelang_ops/
├── cuda/     30 .py   # CUDA TileLang ops
└── asc/      30 .py   # Ascend ops, one-to-one

Sampling, fused sampling, KV read/write, SWA indexing, DFlash, structured output, compression, gather, and norm all have two corresponding copies. The upper layer uses PLATFORM to decide which one to use; model structure, scheduling, and parallel communication are shared.

The underlying kernel libraries are also duplicated as submodules: FlashMLA, DeepGEMM, DeepEP, DeepSelect, plus DeepGEMM-Ascend and DeepEP-Ascend.

To me this is the most practical part of this open-source release for domestic teams. One algorithm logic, two hardware adaptations, and the CUDA/Ascend gap is squeezed as small as possible.

Determinism and structured output

Batch-invariant means: the same input, when placed in a different batch composition, must produce exactly the same result. This is hard because GPU reductions, matrix-multiplication tiling, and FP8 quantization granularity all change with batch shape, so logits under different batch sizes can differ numerically.

Set DEEPINFER_BATCH_INVARIANT=1 to enable it; it currently supports NVIDIA only. tests/models/test_batch_invariant.py verifies it by first generating one trajectory in a batch, then replaying the same trajectory alone and comparing tokens, logprobs, and top-logprobs.

Note that it verifies numerical consistency along a fixed trajectory, not that arbitrary random runs always output the same thing.

Deterministic sampling is stricter: the same input tokens plus the same sampling parameters must produce the same output. Three conditions must hold together:

DEEPINFER_BATCH_INVARIANT=1
DEEPINFER_ENABLE_MTP_VARLEN=0
# and pass a fixed seq_seed to SamplingParams

Variable-length verification is disabled because the dynamic verification length itself changes the execution path.

Structured output uses xgrammar, locked to 0.2.2 in requirements.txt, and supports regular expressions, JSON Schema, and EBNF. It decouples compilation from inference: the schema is first compiled into a grammar automaton, built by a dedicated SO builder process running asynchronously, with DEEPINFER_SO_BUILDER_CPUS controlling concurrency. During inference, each decode step advances the automaton, computes the set of legal tokens, and produces a bitmask applied to sampling.

Near llm_engine.py:913 there is also a guard: if the SO state of the last prefill chunk is wrong, the request is marked dead immediately so it cannot stall the whole scheduling loop.

Three performance charts, and four easy misreadings

All three official charts are useful, but they cannot be used as conclusions in cross-framework comparison.

The first is prefill. The configuration is written above the chart: EP4DP4 | 384+32 experts | context 131,072 | Decoder Bounded Replay (140 tokens). The horizontal axis is time per prefill step; the vertical axis is non-cached input throughput (tokens/s/GPU); the numbers at points are token counts per DP rank.

Going from 4,096 to 32,768, throughput rises from 60,000 to about 87,000. Then from 32,768 to 65,536, throughput only reaches 89,000 while latency grows from 380ms to 730ms.

That is the inflection point of chunked prefill. Pushing batch size further means trading latency for almost zero throughput. The community default of max_num_batched_tokens=32768 matches this inflection.

The second is decode. EP64DP64 | 384+64 experts | context 131,072 | nextn=3 fixed | per-token acceptance 75% -> τ=3.25. The horizontal axis is single-user generation speed; the vertical axis is per-GPU throughput; the numbers at points are concurrent requests per DP rank, not totals.

The conversion:

τ = 1 + nextn × acceptance_rate        # 1 + 3 × 0.75 = 3.25
per_user_tokens_per_s = avg_SPS × τ
tokens_per_s_per_gpu  = avg_SPS × requests_per_DP_rank × τ / CP

Going from 8 to 256 concurrent requests, per-GPU throughput rises from 2,500 to 26,000, while single-user speed falls from 325 to 105. A standard throughput-latency tradeoff.

The third chart overlays EP8DP8 and EP64DP64. At the same single-user speed, EP64's per-GPU throughput is much higher. At 250 tokens/s per user, EP8 is around 6,000 tokens/s/GPU and EP64 is around 15,000.

The reason is not hard to see. Wide-EP spreads experts across more cards; the total weight read per step does not change, but each card reads less. MoE is memory-bound here, so reading less is faster. Add EPLB flattening the load, and TPS goes up. The cost is more cards, a larger communication domain, and harder failure recovery.

There are four misreadings worth calling out separately.

• Routing is controlled. DEEPINFER_FORCE_LOAD_BALANCE=1 combined with Gate's force_random measures a clean performance upper bound, not the hot-spot distribution of real traffic.
• Measurement windows are chosen. Warmup is skipped, intervals that conflict with MoE statistics flush are dropped, and the profiler is turned off during the formal sweep. Each choice is reasonable on its own, but it means the numbers are best steady state, not average experience.
• Requests are constructed. Shared-prefix forks plus uncached suffixes, prefill and decode measured separately, not random HTTP traffic under a cold cache.
• There is no comparison group. The official side did not provide same-config vLLM or SGLang numbers, so these three charts cannot prove DeepInfer is faster than anything; they only show its own scalability.

The decode chart has one more layer: throughput is converted at a 75% acceptance rate, and the script explicitly rejects MTP_VARLEN=1 data to avoid double counting. So it also cannot prove how much DSpark earns on real traffic. For that, look at the table in the official Q&A at 60 concurrency: +5.4%.

What other people think

The most complete source-level analysis is aik8s.run's DeepInfer Architecture and Source Code Deep Dive. That author locked to the first commit d585e65 and read code only, without running GPUs; they say so themselves. Several of its judgments I checked and they hold.

EP64DP64 is not 4,096 cards. Process groups reuse ranks; with CP=1, DP can be defaulted to EP. Zero overhead is a goal. DSpark's gain comes from pricing verification work. YOCO's 140 is suffix replay for a specific path. EPLB's second phase tracks the busiest rank. The official charts have four prerequisites. Quality data is an upstream reference value. Hardware scope is narrower than the promotion suggests: H20, L20, and RTX PRO 5000 Blackwell are not in the official configuration, and the last one is compute capability 12.0, not SM100's 10.0. Building pollutes an existing vLLM environment: it forces torch==2.13.0+cu132 and NCCL 2.32.3, which makes comparison experiments in the same environment pointless.

On the missing KV cache issue, it also explains clearly: cache_engine.py manages the engine's internal KV, including compressed attention, index cache, SWA ring state, and cross-layer reuse, and all of that is present. What is missing is the cross-instance layer: naming, location, transfer, sharing, and lifecycle.

The Hacker News main thread has 2,310 points and 885 comments, but the discussion is mostly about money, not technology. Some calculate that inference is profitable; others remind that inference being profitable does not mean the AI business is profitable, since training and research are still burning cash and models expire. Some worry that open weights plus a pile of inference providers will drive prices through the floor.

One practical feedback item is interesting: V4.1 Flash alone runs at roughly 300 tokens/s, but for agentic tasks it is only 25% to 50% faster, because both the number of thought rounds and the number of commands executed go up.

Chinese media mostly relayed the official press release. The extra number that circulated is an external estimate of gross margin between 82% and 96.94%, depending on cache hit rate and GPU rental prices. It is an estimate, not an official statement.

The boundary of the open source release

This section may be the most practical in the whole post. The gap between what the repository can run and what the architecture diagram shows is bigger than one might think.

Open-sourced	Not open-sourced
Offline LLM.generate() batch inference	OpenAI-compatible HTTP service
C++ scheduler + paged KV + preemption	Cross-instance KV cache storage (3FS)
Single/multi-machine EP, CP, DP	EPD three-stage disaggregated deployment
EPLB expert load balancing	Prefill / Decode / VE three load balancers
DSpark speculative decoding + confidence scheduling	Autoscaling, failover
SWA bounded replay	Prefix router
Multimodal VE + ViT	Request routing across replicas
Structured output	Ops system: metrics, alerting, rate limiting, auth
Dual-backend operators	-

One thing is also easy to overlook. The quadratic cost model cost(q, c) in the official Q&A, which uses a compute budget to back out the chunk size, cannot be found in the repository. I grepped for compute_quota and cost_model; neither exists. It belongs to the Prefill Load Balancer that was not open-sourced.

What actually constrains chunking in the repository are these: max_num_batched_tokens, prefill_max_sum_seqlen_k, the block manager's available block count, and that 0.9 watermark. Do not look for the cost model in csrc/; it is not there.

How I would use it

If your scenario is running V4.1-Flash on SM100 or Ascend 950, and you are doing offline batch inference such as data cleaning, synthesis, evaluation, or batch rewriting, this can be used directly.

If you want to study what a production inference engine looks like, and you want to copy its scheduler and decoding design, it is even more worth reading. It spells out how to price verification work, and that idea can be carried into any speculative decoding implementation.

But if you need an OpenAI-compatible service to serve a frontend or an agent, no. If you need to run Llama or Qwen, no. On H100, H20, or consumer cards, no. If you need multi-tenancy, rate limiting, authentication, or billing, none of that is here.

To fill the gaps, you would at least have to write an HTTP service layer that handles protocol, streaming, cancellation, timeout, and backpressure; do request routing by input length, current decode load, and cache location; add cross-instance KV cache naming, transfer, and lifecycle; add load balancing across replicas; and finally add metrics such as TTFT, TPOT, queue time, and expert load.

There is a fair amount of engineering work.

Getting started

The build environment must be Linux, Python 3.12, and GCC 13 or newer.

export DEEPINFER_ACCELERATOR=nvidia   # or ascend
./build.sh

NVIDIA needs CUDA 13; Ascend needs CANN 9.2. build.sh installs Python dependencies, force-reinstalls NCCL to 2.32.3, and compiles the six vendor libraries and the deepinfer._C extension.

One more reminder: use a fresh Python 3.12 environment. Installing into an existing vLLM environment will change the torch and NCCL versions.

Run the first example:

python examples/offline_inference.py --ep 8

--ep 8 is an example value, not a minimum requirement. The default model is automatically downloaded to the HF cache; alternatively you can point to a local directory with DEEPINFER_HF_MODEL_DIR.

For reading the code, I suggest this order: first look at how examples/offline_inference.py calls it, then entrypoints/llm.py, then dive into step() at llm_engine.py:860. Once you have that main loop, go to scheduler.hpp:275 for scheduling and worker.py:89 for execution. Read speculative decoding separately in mtp_varlen.hpp:133, and see how the model is assembled in v4_modules/builder.py.

Do not read from the first line. This repository is more like a browser rendering engine than business code: the C++ layer is the layout engine, Python is the upper API, and the operator libraries are the compositors.

Finally, my own feeling. The most valuable part of DeepInfer is not yet another inference framework; it is the way a complete production system's co-design is exposed: how verification compute is priced, how expert load is flattened, and how cache flows between stages. In papers these usually appear only as conclusions. Here you can see the code.

As for the 155-trillion number, behind it there are three pieces that were not made public: KV storage, load balancing, and disaggregated deployment. The repository gives you the engine, not the whole vehicle.

---

References:

• DeepInfer official repository
• DeepInfer architecture and source code analysis - aik8s.run
• DeepSeek V4.1 Flash inference system (official blog)
• Hacker News discussion
• DeepSeek-V4.1-Flash-1017 model card

The architecture diagram and three performance curves in this post come from doc/images/ in the DeepInfer official repository, copyright DeepSeek-AI. Source code references correspond to the first public commit d585e65.