Artificial Intelligence · OpenAI

From GPT-2 to Kimi K3: Seven Years of Architecture Upgrades That Go Far Beyond Parameter Count

By 机器之心 · Jul 29, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Kimi K3's architecture is a live catalog of the design constraints that now dominate large-model engineering. Understanding why each component exists, from DeltaNet's write corrections to AttnRes's depth-wise retrieval, is directly useful for anyone building or fine-tuning models that must handle million-token contexts without quadratic compute.

Summary

A 48-hour deep read of Kimi K3's code and eight papers traces every architectural leap from GPT-2's 124M-parameter decoder-only stack to a 2.8T-parameter MoE model. The path runs through linear attention, DeltaNet's targeted memory updates, gated decay, and Kimi Linear's per-channel forgetting, before arriving at K3's hybrid design: Kimi Delta Attention layers for constant-size recurrent state, interspersed Multi-head Latent Attention layers for full-context Softmax retrieval.

K3 also introduces AttnRes, a chunked attention residual that lets later layers selectively pull representations from earlier depths, countering information dilution in the residual stream. The model runs 898 experts (two shared, 896 routed with top-16 selection) in a compressed latent space, uses the SiTU activation, and applies gated MLA to control how much retrieved context enters the forward pass.

The core insight is that every capacity increase targets a specific mathematical bottleneck. Linear attention's cumulative writes cause interference; DeltaNet adds targeted overwrites. DeltaNet can't forget broadly; gating adds decay. Scalar decay is too coarse; Kimi Linear makes it per-channel. Fixed-state memory discards information; MLA and AttnRes retrieve it from context and depth.

Takeaways

— GPT-2's 124M parameters fit into Kimi K3 roughly 22,580 times over, but the architectures share almost no design DNA beyond the residual stream.

— Linear attention replaces Softmax's exponential with ELU+1 applied separately to q and k, enabling a fixed-size D×D state that avoids O(N²) growth at the cost of weaker approximation fidelity.

— DeltaNet adds a per-token write strength beta and subtracts old information before writing new associations, solving the interference problem that pure cumulative linear attention suffers from.

— Gated DeltaNet combines Mamba-2's state decay with DeltaNet's targeted updates, giving the model both precise overwrite and broad forgetting capabilities.

— Kimi Linear makes decay per-channel instead of a single scalar, giving the model fine-grained control over which dimensions of memory persist or fade.

— Kimi K3 alternates Kimi Delta Attention layers with Multi-head Latent Attention layers, so fixed-state recurrent memory is periodically supplemented by full-context Softmax retrieval.

— AttnRes applies learned attention weights across chunks of 12 decoder layers, letting later layers selectively retrieve earlier representations and mitigating residual-stream dilution.

— K3's 898 experts run in a compressed latent space; two are shared across all tokens, and a router picks 16 of the remaining 896 per token, nearly halving FLOPs versus full-dimension experts.

— The SiTU activation replaces SiLU in expert networks, but without fused kernels it runs nearly 3x slower, exposing a recurring inference-optimization bottleneck.

— Every architectural addition from GPT-2 onward solves a specific failure mode of the previous design, not a generic scaling bet.

Conclusions

Scaling laws hold, but the architecture lineage shows that capacity must be added in forms the system can actually exploit. Kimi K3's per-channel decay and depth-wise attention are not just bigger parameters; they are new degrees of freedom that address precise mathematical limits.

The alternating KDA/MLA design is a pragmatic admission that no single attention mechanism is optimal. Fixed-state recurrence is efficient but lossy; Softmax retrieval is expensive but precise. The hybrid approach pays the cost of both but gets the benefits of each where they matter most.

AttnRes's 2% latency penalty for a 1.25x compute advantage is a tradeoff that only makes sense at extreme scale. Smaller models would not amortize the overhead, which is why the technique appears only now.

The SiTU activation's 3x slowdown without fused kernels is a reminder that architecture papers and production inference live in different worlds. A mathematically superior activation is worthless if it cannot be made fast on real hardware.

Concepts & terms

Linear Attention

An attention variant that applies a feature map (e.g., ELU+1) to queries and keys separately before they interact, allowing the K and V matrices to be compressed into a fixed-size D×D state. This avoids O(N²) memory growth but provides a weaker approximation of the Softmax kernel.

DeltaNet

A recurrent architecture that improves on linear attention by computing a per-token write strength (beta) and subtracting the existing value at a key before writing a new association. This prevents the interference that occurs when purely cumulative writes saturate a fixed-size memory.

Gated DeltaNet

Combines DeltaNet's targeted overwrite with Mamba-2's state decay. A learned scalar alpha (0 to 1) controls how much of the old state persists, giving the model both precise memory updates and the ability to broadly forget when context shifts.

Kimi Delta Attention (KDA)

Kimi Linear's evolution of Gated DeltaNet that makes the decay factor per-channel instead of a single scalar. Each dimension of the hidden state gets its own learned forgetting rate, enabling fine-grained memory management.

Multi-head Latent Attention (MLA)

A Softmax-based attention variant that operates in a compressed latent space. In Kimi K3, MLA layers are interleaved with KDA layers to periodically retrieve information from the full token context, compensating for the information loss inherent in fixed-size recurrent states.

AttnRes (Attention Residual)

A chunked attention mechanism applied every 12 decoder layers that computes learned weights over earlier residual-stream states. It lets later layers selectively retrieve representations from earlier depths, countering information dilution and hidden-state magnitude growth in deep residual networks.

SiTU

An activation function used in Kimi K3's expert networks, defined as beta * tanh(x/beta) * sigmoid(x). It replaces SiLU but can run nearly 3x slower without fused GPU kernels, making it a practical inference bottleneck despite its mathematical properties.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗