跪拜 Guibai
← Back to the summary

The Architecture Terms Behind China's Latest Open-Source LLMs, Explained Plainly

theme: smartblue

Recently, GLM 5.3 Flash, Qwen 3.8 Flash, and Hy4 Preview have all been released in a cluster. Many people read the terminology introductions and think, "What the hell is this?" So let's just briefly chat about what these things are. The main goal is to at least understand what these things are and know what they're useful for.

For example, in the recently released Hy4 preview content, there's MoE, Top-8, Shared Expert, GQA, MLA, DSA, Indexer, IndexCache, Residual Connection, iHC, MTP, Speculative Decoding...

First, the most basic and common simple terms are:

Term What it is Function Example
Parameter Numbers produced by model training; can be understood as the place where it stores knowledge and computation rules More parameters generally mean larger potential capacity, but also higher VRAM, communication, and deployment costs 770B means approximately 770 billion parameters
Activated Parameters Parameters that actually participate in the computation when processing one Token MoE can retain huge total capacity while avoiding running the full model for every Token Hy4 totals 770B, but only activates about 49B per Token
MoE Mixture of Experts, splits the FFN into many parallel experts, only calling a portion each time Increases model capacity while controlling the computation per Token Hy4 has 256 routed experts per MoE layer
Router A very small network that scores all experts Decides which experts the current Token is sent to A Token gets the highest scores for experts 17, 43, 91, etc., and is then sent to those experts
Top-8 Selects the 8 highest-scoring experts from 256 routed experts Makes each Token only compute a few experts For example, the Token "Flutter" selects 8 experts in one layer, and might select another group in the next layer
Shared Expert An expert that every Token must pass through Handles general knowledge needed for all types of input, reducing the need for routed experts to re-learn basic capabilities For example, general processing like grammar, basic language understanding can go into the shared expert
Dense FFN All Tokens go through the same complete set of FFN parameters Computation is stable, no expert routing issues, but larger parameters mean more expensive computation Hy4 uses Dense FFN in the first layer, and MoE in the following 77 layers

Also, "expert" is just a structural name; it cannot be directly understood as a "code expert" or "finance expert". After training, some division of labor may emerge, but it's not guaranteed that these 256 experts can be labeled with human-readable tags.

Finally, there's the weight size, for example Tencent Hy4's 770B/49B. Here, the 770B weights all need to be placed into GPU VRAM or memory. A rough calculation: 770B of BF16 weights requires about 1.54 TB (770B × 2 Byte ≈ 1.54 TB). If it's FP8, it's close to 770GB. Of course, this doesn't yet count MTP, KV Cache, and runtime buffers. Also, Hy4 uses a Top-8+1 shared expert structure.

Next are some parameter-related terms, like:

Term What it is Function Example
Layer A computational unit where the model performs one round of Attention and FFN processing The number of layers reflects processing depth Hy4 has 78 layers; a Token's state must go through 78 consecutive updates
Hidden Size The length of the vector used to represent each Token inside the model A wider vector can encode more features simultaneously, but also increases computation and VRAM overhead A Token in Hy4 is generally represented by 6144 numbers
Context Length The maximum number of Tokens that can be placed in a single request Determines how much code, documentation, and conversation history the model can process at once 1M can accommodate large codebases, logs, and multi-turn tool results

Next are some more unfamiliar keywords, like GQA, MLA, DSA, and IndexCache. Here we need to first briefly understand ordinary Attention. Generally speaking:

When the model generates the current Token, it takes the current Token's Query, compares it against the Keys of historical Tokens, and then retrieves the corresponding Values.

It can be simplified and understood as:

Attention Terminology

Then here's a simple understanding of these terms:

GQA

GQA is Grouped Query Attention, where multiple Query attention heads share the same set of Keys and Values.

For example, with 64 Query heads, ordinary Multi-Head Attention might need to save 64 sets of K/V, while GQA can let every 8 Query heads share one set, saving only 8 sets of K/V. It mainly reduces KV Cache and memory bandwidth, making it more suitable for long-context inference.

Hy3 used this kind of relatively conventional structure.

For instance, ordinary MHA is like giving 64 investigators each a complete copy of the historical archives. GQA's approach is to have every 8 investigators share one set of archives, so ultimately only 8 sets need to be saved.

However, the questions the investigators want to ask might still differ, so there are still 64 Queries. It's just that the Key/Value archives they consult can be shared. That is, 64 questions are still 64 questions, just saving many fewer copies of historical K/V.

MLA

MLA is Multi-head Latent Attention. It further compresses K/V into shorter latent vectors, which participate in attention computation when needed.

Hy4's KV compression dimension is 512. Simply put, it can compress the originally massive multi-head K/V information of each historical Token into a 512-dimension "compressed package". It mainly reduces KV Cache usage and read bandwidth for 1M context.

Strictly speaking, MLA is closer to the model's attention structure itself learning a low-dimensional latent representation. When attention computation is needed, it constructs/participates in the corresponding calculation from this latent representation.

Intuitively, it can be understood as: ordinary Attention saves multiple complete sets of K/V for each historical Token, while MLA does not directly save such a wide set of information long-term. Instead, it chooses to learn a more compact latent state.

For example, Hy4's KV compression dimension is 512. It can be crudely understood as each historical Token leaving behind a 512-dimension "compressed archive", which is then used to participate in attention computation when needed later.

So both GQA and MLA reduce K/V costs, but MLA's compression intensity and structure are more aggressive. It's similar to:

DSA

DSA is DeepSeek Sparse Attention. It mainly solves another problem:

Even if K/V has been compressed, if the current Token still pays attention to the previous 1 million Tokens one by one, the computation load is still very large.

So DSA adds a lightweight Indexer:

For example, the context contains a Flutter repository with 1 million Tokens, build logs, and Issues. Then the current question is:

Why does firebase_auth fail to compile under Kotlin 2.4?

The Indexer might pick out from a million positions:

The main Attention then focuses on processing these positions. DSA reduces the core attention complexity from approximately O(L^2) to O(Lk), where k here is 2048.

However, it should be noted that the Indexer itself still needs to scan the historical information, so there is still a cost, just much lighter than the main Attention.

Gated DSA

Gate is a "valve". From the Hy4 configuration's gated_mla=true, gating_type=elementwise, and the linear_gate in each layer, it performs element-wise control on the attention output, then decides how much of the information retrieved this time should enter the subsequent state.

Simply put, the Indexer is more inclined to decide "where to look", and the Gate is more inclined to regulate "how much of these features already produced by Attention should be written back to the subsequent state".

So the Gate can also be understood as many "small volume knobs" after the attention output. The Indexer has already decided which historical positions to focus on this time, and Attention has already aggregated this information back. The Gate can then control, element by element, the strength with which these results enter the subsequent state.

For example, the Attention output simultaneously forms different internal features like "This is a Kotlin compilation error", "Involves FirebaseAuth", "Might be related to dependency versions". The Gate can suppress the parts unimportant for the current step and retain the more useful features.

IndexCache

Because theoretically, every layer of DSA needs to re-run the Indexer and re-select 2048 positions, but the positions selected by adjacent layers are generally highly similar. So IndexCache divides the 78 layers into:

Hy4's pattern is:

0, 1 layers compute independently
2, 3, 4 layers reuse
5 layer recomputes
6, 7, 8 layers reuse
9 layer recomputes
……

Here, a total of 21 layers run the full Indexer, and 57 layers reuse it, so roughly 73.1% of layers save on independent Indexer computation. But this is not reusing attention results; it only reuses the list of "which historical positions should be looked at".

Previous papers on IndexCache mentioned removing 75% of Indexer computation on a 30B DSA model, reporting up to 1.82x Prefill acceleration and 1.48x Decode acceleration.

It can be simply understood as: Layer 20 just made a list of "the 2048 pages most worth reading" from 1 million pages of material. Then layers 21, 22, 23 find that their focus is usually similar to layer 20's, so they directly borrow this book list for their Query.

Residual Connections, Residual Stream

Generally, each layer of an ordinary Transformer is roughly xl+1=xl+F(xl), where xl is directly preserved, and F(xl) is the new information computed by this layer. This path that directly preserves the original state is the residual connection or Identity Path.

Its function is to prevent the original information and gradients from being easily lost as the model gets deeper.

That is, for example, the Token state received by layer 20 already contains "This is a Flutter compilation problem", and then layer 20 analyzes a new piece of information "The problem might be related to Kotlin type inference". Without a residual connection, it can be crudely understood as layer 20 directly overwriting the old state with the new result. But with a residual connection, it's similar to:

Old notes + New annotations from this layer = New notes handed to the next layer.

So as the model goes layer by layer upwards, it doesn't need to re-preserve the previous information from scratch at every layer.

Residual Stream

The residual stream can be understood as the internal state channel running through all layers. An ordinary Transformer has only one. Both Attention and FFN read from here and write their results back here. For example, the current Token's 6144-dimension state is updated by layer 20, then handed to layer 21 for continued processing.

The Residual Stream is this "shared notebook" that is passed from layer 1 all the way to layer 78, constantly being appended to and modified.

Hyper-Connections

Hyper-Connections expand one residual stream into multiple streams, then let each sub-layer dynamically decide:

This can increase the information transmission bandwidth between layers.

iHC

iHC is identity Hyper-Connections. Hy4 maintains four parallel residual streams for each Token, each stream being 6144 dimensions. A single sub-layer computation roughly goes:

For example, read weights might be:

[0.5, 0.2, 0.2, 0.1]

Write-back weights might be:

[1.0, 0.1, 0.7, 0.4]

The four streams will gradually form different states. Although it can be crudely understood as "one preserves semantics, one preserves task state, one preserves tool results", this is just an example to aid understanding. In reality, the model has no such fixed division of labor.

The Identity in iHC means the four old streams continue on their own paths, without using a 4×4 matrix to mix them together. Essentially, the input is copied into four streams, and the sub-layer dynamically Reads/Writes.

That is to say, an ordinary Transformer is like a company having only one public notebook. Every layer reads from this one book and then writes its own results back into it. As the model gets deeper, important early information may continuously get stirred together with later content.

iHC, on the other hand, is like maintaining 4 parallel notebooks simultaneously. When a certain layer works, it can say, "This time, mainly refer to notebook 1 and notebook 3", and after calculating, decide "Write the results mainly back to notebooks 2 and 3, only write a little to notebook 1". Thus, when a Token passes through 78 layers, it no longer has just one information highway but can have four pathways simultaneously.

mHC

mHC also has multiple residual streams, but it additionally learns an inter-stream mixing matrix. The old first stream can be mixed into the second and third streams, and then constrained by a doubly stochastic matrix and Sinkhorn projection to prevent training from going out of control.

So its expressive power is stronger, but computation, VRAM access, and training complexity are also higher.

If iHC is "4 notebooks each retain their own old content, only allowing the current sub-layer to take information from a few of them and write back separately", then mHC additionally allows, before entering the next layer, directly moving part of the old content from notebook 1 into notebooks 2 and 3.

This way, the communication ability between the four streams is stronger, but it also easily leads to the problem of one notebook becoming stronger and stronger while others get diluted. So mHC needs extra constraints on this cross-stream mixing matrix. For example, the role of Sinkhorn is "Don't let all the information crowd into just one highway."

Qwen GR

Here we can also additionally introduce Qwen3.8 Flash Next's GR, which is Gated Residual. It also expands the residual stream into four pathways and dynamically controls reading and writing back. Qwen officially describes it as element-wise Read Gate and per-branch Write Gate.

So GR and iHC both belong to the same round of "multi-path residual stream" exploration, but the implementation details differ, though the structural idea is similar.

Here, Qwen officially also provided an observation very suitable for popular science:

After training the four-path residual stream, one of the paths naturally forms a more obvious long-range pathway, bringing information from earlier layers to deeper layers.

This is like building four roads for the model. The engineers didn't specify which one is the highway, but after training, one naturally emerged as a main artery more suitable for long-distance transport.

AttnRes

Generally, Attention can not only look at previous Tokens but also "look at previous layers". When introducing iHC, mHC, and Qwen GR earlier, they were mainly solving one problem:

As the model gets deeper, how does information stably transmit from layer 1 to layer 80 or 90?

Kimi K3 uses Attention Residuals, or AttnRes, and the idea is a bit different.

Ordinary Residual can be crudely understood as: each layer adds its new results into the same notebook. So when the model reaches layer 80, what it holds is a large notebook accumulated from dozens of previous layers. The problem is that each layer fixedly adds x = x + F(x), meaning:

Going further back, the content in this notebook becomes more and more. Very valuable information from an early layer may gradually be diluted by a large amount of subsequent updates.

AttnRes simply chooses to let the current layer judge for itself: Which representations left by previous stages do I actually want to refer to more right now?

In implementation, it performs Attention on the representations of previous layers or Blocks. So it's not that all historical layers are always accumulated into the Residual with a fixed +1. AttnRes dynamically allocates weights based on the current content. For example, the model has already processed up to layer 80 and is making the final judgment:

What exactly is the reason for the firebase_auth compilation failure?

It might discover:

AttnRes can then dynamically:

Take a bit more information from layers 20 and 40, and a bit less from other stages.

So here there is actually a very interesting distinction:

To avoid saving all the outputs of all 93 layers, Kimi K3 actually also uses Block AttnRes, grouping multiple layers into one Block, and then performing this cross-depth retrieval at the Block level. So now, looking at the Residuals of several companies together is very interesting:

Architecture Simplest Understanding
Ordinary Residual One notebook written from beginning to end
iHC / GR One notebook expanded into multiple parallel notebooks
mHC Multiple notebooks can also dynamically exchange content with each other
AttnRes The current layer can actively flip through the notebooks of previous stages

Linear Attention and Hybrid Architectures

The GQA, MLA, and DSA discussed earlier essentially all revolve around one problem: How can the model more cheaply find things from very long historical Tokens? Then if you look at the recent GLM 5.3 Flash, Qwen3.8 Flash Next, Kimi K3, there are some other terms: Linear Attention, KDA, GDN, QSA...

This is because many models now no longer intend to have every layer save and consult the complete history. Ordinary Attention can be understood as: every time a new Token is read, the previous materials are kept in the filing cabinet, and when needed, they are looked up via Query / Key.

If the context is only a few thousand Tokens before, doing this is indeed no problem. But when it reaches 1 million Tokens, even if KV has been compressed with MLA and fewer positions are looked at with DSA, the model still needs to maintain a very large set of historical information.

So another route is: Can we, while reading, continuously compress the history into a fixed-size "working memory"? This is the most intuitive understanding of the Linear Attention route.

KDA and GDN

For example, the model is now reading a Flutter repository with 1 million Tokens. Ordinary Attention would try to keep the materials of the previous 1 million Tokens as much as possible, and look back when needed later.

But if it's Linear Attention, it would be similar to: "Every time I read a page, I continuously update a 'working notebook' in my hand, and later I mainly continue reading with this notebook."

So Linear Attention does not need the KV Cache to keep linearly expanding as the context gets longer. Then the two more common names currently are:

KDA itself is a Linear Attention further developed along the lines of Gated DeltaNet. The core is maintaining a fixed-size recurrent state, meaning the internal state is continuously updated as Tokens arrive. Then KDA makes the control of memory decay more fine-grained, allowing different internal channels to have different forgetting speeds.

It can be crudely understood as: some information like "This is a Flutter project" might need to be remembered for a long time, while some information like "The current value of that variable just now is 17" might be useless after a few steps.

So KDA is not just "continuously adding things to the notebook", but also needs to learn 'what should be kept, what should be overwritten, what should be slowly forgotten'.

However, Linear Attention also has a cost. Because you are continuously stuffing hundreds of thousands of Tokens into a fixed-size state, you are essentially doing continuous information compression. Some very precise historical details may become increasingly difficult to recover. So what is truly popular now is actually:

Hybrid architectures of Linear Attention + Ordinary/Sparse Attention.

For example, Qwen3.8 Flash Next is a very typical example. It cyclically uses 3 layers of GDN + 1 layer of QSA:

Similarly, Kimi K3 follows a similar idea, 3 layers of KDA + 1 layer of Gated MLA continuously cycling:

KDA is responsible for low-cost maintenance of long sequence states, and the periodic MLA is responsible for truly doing global Token-to-Token Attention.

GLM 5.3 Flash has also taken a similar route, mixing Linear Attention and Sparse Attention together:

So simply put, this kind of hybrid architecture can now be understood as:

Usually write summaries; when details are needed, flip through the original materials.

QSA, IndexPool: Now even the Indexer itself is starting to be compressed

Earlier when talking about DSA, it was said that the whole process is similar to:

1 Million Tokens - Indexer broad selection - Find the most relevant 2048 - Main Attention precise calculation.

But doing this leads to another problem: The main Attention is saved, but the Indexer itself still needs to scan 1 million positions.

So a common direction among several recent players is to start compressing even this "broad resume screening" step. For example, Qwen3.8 Flash Next's QSA, which is Qwen Sparse Attention, does this:

It doesn't let the Indexer directly do a complete index on every single Token. Instead, it first aggregates consecutive Tokens into coarser micro-blocks. The Indexer first judges which few regions are most worth looking at? Then these regions are handed over to the subsequent Sparse Attention.

If we continue using the resume-finding analogy, DSA is similar to:

Quickly scan all 1 million resumes, then pick 2048 people.

And QSA becomes:

First organize the resumes into many booklets by department, project, or team. First judge which booklets are worth looking at, then go inside to find specific people.

So QSA saves not just Attention; it also shrinks the Indexer's search space.

GLM 5.3 Flash's IndexPool is also solving a similar problem, just with a different implementation method. GLM compresses 4 Indexer Keys into 1 through weighted Pooling. So the index information that the Indexer originally needed to maintain and read is directly reduced by roughly 4 times, and only then is the subsequent Top-k executed.

Simply put:

One is compressing "how many things to search this time", the other is reducing "how many times to search the same thing".

DeepSeek V4's CSA / HCA: Compress the history first, then decide how to search

Then on this issue, DeepSeek V4 seems to have gone a step further. It now mainly alternates between two types of Attention:

CSA can be simply broken into two parts. First, it compresses KV along the Token sequence:

Approximately every 4 historical Tokens → 1 compressed KV Entry.

For example, originally there are 1 million historical positions, first compressed into about 250,000 compressed positions. Then the Lightning Indexer performs Top-k from these already compressed positions. Finally, Sparse Attention only processes the selected part, while additionally keeping a small Sliding Window to preserve fine-grained information of recent Tokens.

So compared to the earliest DSA:

In this case, the Indexer itself naturally becomes much cheaper.

Then HCA is even more aggressive. It roughly:

Compresses every 128 Tokens → into one historical Entry.

This way, 1 million Tokens end up as only seven or eight thousand compressed blocks. After compressing to this extent, DeepSeek simply doesn't even need the Indexer anymore, directly performing Attention on all these highly compressed historical blocks.

That is, CSA is 'compress a bit → then Top-k', and HCA is 'compress very aggressively → look at everything'.

DeepSeek V4 alternates between these two types of layers. One is better at precisely selecting key points from history, the other is more like cheaply obtaining a broad overview of the entire history.

So Looking Back at These Architecture Diagrams, It's Actually Not That Confusing

These new models are mainly simultaneously modifying several "information highways":

Problem Current Common Approaches
How to remember 1M Tokens cheaply KDA / GDN type Linear Attention
What to do when precise historical info is needed MLA / Sparse Attention / QSA / CSA
What to do when even the Indexer is too expensive IndexPool / compressed indexer
What to do when the historical KV itself is too large MLA / CSA / HCA
Model too deep, early info easily fades iHC / mHC / GR / AttnRes

Finally, if we go back to the previously posted diagrams, we can probably simply understand the common terminology implemented in all current domestic open-source large models:

preview

preview

Also, using a diagram from an expert to explain KV Cache: Why does the KV cache store K and V vectors, but not the Q vector we kept mentioning earlier?

Simply put, although each token produces its own Q, K, V vectors, Q is a use-and-throw-away thing. Caching it is a complete waste of memory. A new token only needs its own Query to "query" the history. The historical Keys and Values are fixed and reusable. So KV Cache only stores K and V. Each time, only the Q of the current new token is calculated (along with its own K/V, which are then appended to the cache):

Speculative Decoding and Drafts

This part has actually been discussed before. For details, see "vLLM Testing Different AI Speculative Decoding Architectures". Simply put, even if only one new token is added per step, the model weights need to participate in a complete forward computation. Speculative decoding first lets a cheaper draft component guess some future tokens, for example guessing 4 at a time. Then the target model checks these 4 positions in one verification pass. If the first 2 match the target model's distribution and the 3rd is rejected, this round can submit 2 draft tokens at once, thus speeding things up considerably.

img

Then different speculative decoding frameworks have the same goal but different implementations. For example, MTP is Tencent Hy4's default speculative decoding. Hy4 has one built-in MTP layer, with total parameters 10B, activated 0.7B, trained together with the main model and packaged inside the model.

Other common ones include EAGLE-3, DFlash, and DSpark.

Quantization

This area is probably one of the easiest places for ordinary people to get confused about in large models, because you often see simultaneously:

BF16, FP8, MXFP8, INT4, W4A16, GPTQ, AWQ, Q4_K_M, IQ2_XXS...

It all looks like "quantization", but it's hard to intuitively judge from the text alone what they all are. Actually, it can be most simply divided into three layers first:

What you see What it mainly talks about Example
BF16 / FP8 / INT8 / INT4 How a single number itself is represented Does each weight cost 16 bit, 8 bit, or 4 bit
MXFP8 / NVFP4 / W4A16 How a bunch of low-precision numbers are organized and used Every 32 numbers share one Scale; Weights 4-bit, Activations 16-bit
GPTQ / AWQ / Q4_K / IQ Specifically how to compress the model down Which errors to protect, how to encode each block of weights

So the core purpose of quantization is actually very simple:

The model originally has hundreds of billions of numbers. Can we spend fewer bits to save these numbers, while trying not to crush the model's capability.

For example, Hy4 has 770B parameters. If all weights use BF16, one parameter is 2 Bytes. Just the weights theoretically are:

770B × 2 Byte ≈ 1.54 TB

If on average it can achieve 8 bit, it's about 770GB. Achieve 4 bit on average, theoretically around 385GB.

Of course, real model files will also have Scales, Metadata, high-precision layers, and other extra content, so you can't directly take this number as the final file size. But the general direction is: Fewer bits, smaller model, lower memory bandwidth pressure.

Starting from BF16, FP16, FP8

As is well known, computers cannot save a decimal with infinite precision. So floating-point numbers divide the limited bits into three parts:

Sign bit + Exponent + Mantissa

It can be crudely imagined as a ruler:

So even though both are 16 bit, the "personalities" of FP16 and BF16 are actually quite different:

Format Bit Layout Biggest Feature Common Use
FP32 1 + 8 + 23 Large range, fine markings, but takes up space High-precision computation, accumulation, some optimizer states
FP16 1 + 5 + 10 Markings relatively fine, but range relatively narrow Inference, traditional mixed-precision training
BF16 1 + 8 + 7 Range close to FP32, but markings coarser Very common in modern large model training and inference
FP8 E4M3 1 + 4 + 3 Smaller range, relatively higher precision Weights, activations, etc.
FP8 E5M2 1 + 5 + 2 Larger range, lower precision Scenarios requiring higher dynamic range

So why has BF16 been particularly common in recent years? Mainly because it gives all 8 bits to the exponent, same as FP32, so it can represent very large or very small numbers.

FP16 gives more space to the mantissa, so decimals near the same range are divided more finely. But it only has 5 exponent bits, so the range it can cover is noticeably narrower. For example, some gradients during training might be particularly small.

So in this case, FP16 is similar to:

My ruler's markings are quite fine, but this number has already become too small to fit within my measuring range.

Then BF16 is more like:

My markings aren't as fine as yours, but at least I can still measure this number.

So in large model training, many times "first avoid overflow or underflow" is more important than keeping a few extra decimal places.

Then FP8 is continuing to push down, using only 8 bits per number. This way, theoretically, weight storage and memory read volume can both be nearly halved compared to BF16. However, the numbers FP8 can represent are noticeably more limited, so in actual use, a very important problem starts to arise: Scale.

What is Scale for?

For example, when I want to squeeze the following numbers into a low-precision format:

0.02, 0.03, 0.05, 0.08, 18.0

The problem is that most numbers are around 0.0x, but a 18.0 is mixed in. If all numbers directly share the same limited range, then to accommodate 18.0, the entire range must be stretched very large. The result is that 0.02 and 0.03 might end up being compressed into similar values.

So many low-precision formats assign a Scale, i.e., a scaling factor, to a group of numbers. Simply understood as:

I first look at roughly how big this group of numbers is, then decide what measuring range ruler to give them.

For example, the original number is 18, which the low-precision format can't hold. So it can first be divided by a Scale: 18 ÷ 8 = 2.25. Then save this 2.25, and multiply it back when calculating later.

So in actual low-precision computation, it's definitely not simply a single number directly chopped to 8 bits. It's actually a group of low-precision numbers + a Scale. This is also why the final average storage of a so-called "4-bit model" is often not strictly 4.000 bit/weight, because the Scale itself also takes up space.

Then what is MXFP8?

After having Scale, there's another problem: Exactly how many numbers share one ruler?

For example, if hundreds of thousands of numbers all share one Scale, as long as a few particularly large outliers are mixed in, the other small numbers will still be squeezed uncomfortably. So MX, which is OCP Microscaling, simply chops the Tensor into very fine pieces.

For example, MXFP8 typically has every 32 elements share one Scale. It can be imagined as: originally, hundreds of thousands of people in a stadium shared one ruler. Now it's changed to:

Every 32 people get their own ruler.

This way, even if each element itself only has 8 bit, 6 bit, or even 4 bit, it's easier to preserve the original information. OCP's MX formats can roughly be viewed like this:

Name Per Element Group Size Scale Approx Effective Storage
MXFP8 FP8 E4M3 / E5M2 32 E8M0 ~8.25 bit
MXFP6 FP6 E2M3 / E3M2 32 E8M0 ~6.25 bit
MXFP4 FP4 E2M1 32 E8M0 ~4.25 bit
MXINT8 INT8 32 E8M0 ~8.25 bit

Here, the meaning of E8M0 can be simply understood as:

The Scale itself is also a very compact floating-point number, and it only represents powers of 2.

So although MXFP4 is called 4-bit, the actual average is about 4.25 bit, because an extra Scale needs to be saved for every 32 elements.

Then what's the difference between MXFP4 and NVIDIA's NVFP4?

NVFP4 is also 4-bit floating point, but NVIDIA made the Scale finer on Blackwell. The numbers inside the blocks on both sides are themselves E2M1:

1 sign bit + 2 exponent bits + 1 mantissa bit

The real difference is mainly in the Scale:

MXFP4 NVFP4
Block Size 32 elements 16 elements
In-block numbers E2M1 E2M1
Local Scale E8M0, can only scale by powers of 2 FP8 E4M3, scaling is finer
Extra Scale None Usually also has tensor-level FP32 Scale
Average Storage ~4.25 bit ~4.5 bit

That is, MXFP4 has 32 people sharing one ruler that can only be adjusted in fixed steps, but NVFP4 has 16 people sharing one ruler with finer adjustments. So although NVFP4 spends a little more space on average, it's usually easier to get close to the original numbers, and precision is better.

So by this point, BF16, FP8, MXFP8, NVFP4 this group is already relatively clear:

Then there are words like INT8, INT4, W4A16

The previous BF16, FP8, FP4 are all still floating-point numbers. Another major category of quantization directly uses integers, for example:

Of course, the model's original weights are definitely not just these integers, so it still relies on Scale. For example, the original weight 0.137, after quantization might be saved as 7, while recording Scale = 0.02. This way, when restored, it's roughly 7 × 0.02 = 0.14. That is, the original 0.137 became 0.14. This is quantization error.

The truly troublesome part of model quantization is also this:

How to prevent billions, tens of billions, or even hundreds of billions of these small errors from collectively ruining the model's capability in the end.

Then you will see names like W4A16, W8A8. Here:

So W4A16 means Weights are 4-bit, and Activations are still 16-bit during runtime. Its biggest benefit is that the model weights are much smaller, while Activations retain relatively high precision, so it's also a common approach for many local models.

And W8A8 means both Weights and Activations are 8-bit. W4A8 means Weights 4-bit, Activations 8-bit.

Then what are NF4, GPTQ, AWQ, SmoothQuant?

Here we reach yet another layer. The previous INT4, FP4 were mainly talking about:

How a number is ultimately represented.

Then the purpose of GPTQ, AWQ, SmoothQuant is: How do I turn the model into these low-precision numbers while losing as little capability as possible.

For example, now there are ten thousand weights to compress to 4-bit. The crudest method is of course rounding them all together. But weights in a model are not equally important. Changing some weight 0.81 → 0.80 has almost no impact, but slightly altering some sensitive positions, passed down layer by layer, might change the output a lot. So various quantization algorithms emerged:

Name How it can be understood
NF4 A set of 4-bit numerical representations specifically designed for approximately normally distributed weights, very common in QLoRA / bitsandbytes
GPTQ A PTQ quantization algorithm that uses approximate second-order information to try to compensate for the output error caused by weight quantization
AWQ Observes Activations, focusing on protecting weight channels that are more important to the model's output
SmoothQuant "Moves" the outlier parts of Activations that are particularly hard to compress to the Weights, making W8A8 easier to do
PTQ Post-Training Quantization, quantizing the model after training is finished
QAT Quantization-Aware Training, letting the model adapt to the future low-precision environment during the training phase

So it can be simply understood as:

4-bit just tells you how much budget is left. Names like GPTQ, AWQ tell you how to spend these 4 bits.

What are Q4_K_M, IQ3_XXS in GGUF?

If you usually use llama.cpp, LM Studio, Ollama, or download GGUF from Hugging Face, what you see will be another set of names:

Q4_0
Q4_K_M
IQ3_XXS
Q8_0
UD-Q4_K_XL
MIX-STQ1_0

Here, the first thing to note is that these filenames are more a set of community quantization recipe names, and cannot be simply understood as a certain standard 4-bit number format. For example, Q4_K_M.gguf does not mean that all Tensors in the entire model are strictly 4-bit. Actually, it more represents a quantization preset.

The publisher can make most of the model's weights use low bits, while keeping certain more sensitive Tensors at higher precision. So looking at these names, I think the simplest method is still to first look at the number in front:

Q2 → Q3 → Q4 → Q5 → Q6 → Q8 → F16/BF16

But the number here can only be understood as "roughly in the several-bit tier", because besides the weights themselves, Scale, Min, codebooks, and certain high-precision Tensors all take up space.

For example, the underlying Q4_K Tensor itself is roughly 4.5 bpw, but a complete Q4_K_M.gguf might mix other precisions inside, so the entire model's average bpw can be higher. Therefore, don't see Q4 and directly take parameter count × 4 bit as the final file size.

What are _0, _K, IQ?

Are you almost dizzy from reading? Actually, these are mainly about which set of encoding scheme is ultimately used. For example:

Name Simple Understanding
Q4_0 A relatively early set of 4-bit block quant in llama.cpp
Q4_1 Additionally saves some range information on top of the Q4_0 idea
Q4_K K-quant, uses larger super-blocks, while further compressing auxiliary data like Scale
IQ4_XS I-quant, uses more complex codebooks and importance processing
TQ1_0 Ternary quantization, weights roughly only left as -d / 0 / +d
STQ1_0 Sparse Ternary, further adds a fixed sparse structure on top of ternary

So when downloading GGUF normally, you can roughly view it like this:

_0 / _1 are often earlier block quants. _K is a very common K-quant family currently in llama.cpp. IQ usually uses more complex encoding in exchange for better ultra-low bit performance. TQ / STQ already belong to very aggressive ternary, sparse ternary quantization.

So actually, there's no need to get tangled up over what the letter _K exactly represents. Knowing it's a K-quant / super-block quantization family is enough.

How to view _S / _M / _L / _XL?

Then this is another confusing thing. They can be roughly understood as:

Different size and quality-oriented recipes under the same quantization family.

For example, some schemes will:

So usually, the more the suffix leans towards "large", the larger the file will be, and the quality more conservative. But it should be noted:

S / M / L / XL are not a unified set of levels that all quantization methods must support as stipulated by GGUF.

For instance, llama.cpp itself has Q3_K_S / Q3_K_M / Q3_K_L, but for Q4_K, the common ones are Q4_K_S / Q4_K_M. Then the standalone Q4_K in current llama.cpp is actually also an alias for Q4_K_M.

And as for the _XL in UD-Q4_K_XL, it's more of an extra tier added by quantization schemes like Unsloth Dynamic themselves. So you can roughly use S < M < L < XL to judge "this publisher is probably increasingly preserving quality", but exactly which Tensors were upgraded by how many bits still depends on the specific quantization recipe.

What about prefixes like UD-, MIX-?

This is relatively simpler. They are usually markers used by the publisher to indicate their own mixed quantization schemes, for example:

Prefix Rough Meaning
No special prefix Often directly uses llama.cpp's quantization preset names
UD- Unsloth Dynamic, dynamically selects quantization types based on the sensitivity of different Tensors in the model
MIX- A set of Mixed Quantization combined by the publisher themselves

So a UD-Q4_K_XL expresses: This is an overall volume roughly at the 4-bit tier, quality-oriented, Unsloth Dynamic mixed quantization scheme.

For example, Hy4's 200GiB version. Hy4 originally has 770B parameters, BF16 weights theoretically about 1.54 TB. But now there is already a MIX-STQ1_0 GGUF just over 200GiB. What the official team did is a very aggressive mixed quantization:

In the end, the entire model averages down to around two or three bit/weight.

Interested readers can see: Hy4 compresses a 1.5TB model into a 200GiB GGUF MIX-STQ1_0 custom version, with almost no performance loss.