Artificial Intelligence

30 More LLM Concepts, Illustrated: From KV Cache to Mamba

By 周末程序猿 · Jul 11, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

LLM jargon now saturates every layer of the stack, from GPU kernels to alignment research. A single-sheet visual reference for 30 terms cuts through the noise and gives engineers a shared vocabulary for discussing throughput bottlenecks, model architecture, and safety tuning.

Summary

The second installment of a three-part visual dictionary maps out 30 more essential LLM concepts, numbered 31 through 60. Each entry pairs a clean diagram with a concise definition, moving from foundational mechanics like the KV Cache and Multi-Head Attention to efficiency techniques such as FlashAttention, quantization, and speculative decoding.

Later entries shift toward architectural alternatives and post-training alignment. Selective State Space Models (Mamba), Mixture of Experts (MoE), Direct Preference Optimization (DPO), and Constitutional AI all appear, alongside practical inference strategies like continuous batching and guided generation. The series functions as a rapid-reference map for anyone navigating the increasingly dense LLM toolchain.

Takeaways

— KV Cache stores previously computed Key and Value vectors to avoid recomputing them at each decoding step, drastically speeding up autoregressive generation.

— Multi-Query Attention (MQA) shares a single Key-Value head across all Query heads, reducing the memory footprint of the KV Cache.

— Grouped-Query Attention (GQA) splits Query heads into groups that each share one Key-Value head, balancing MQA's memory savings with Multi-Head Attention's quality.

— FlashAttention fuses attention operations into a single GPU kernel and tiles the computation to minimize high-bandwidth memory reads, achieving significant speedups and memory savings.

— PagedAttention, used in vLLM, manages the KV Cache in non-contiguous blocks to eliminate memory fragmentation and improve GPU utilization during serving.

— Continuous Batching appends new requests to a running batch at each iteration instead of waiting for the whole batch to finish, raising serving throughput.

— Speculative Decoding uses a small draft model to propose several tokens quickly, then has the large model verify them in parallel, producing identical output with lower latency.

— Quantization maps high-precision model weights and activations to lower bit-widths (INT8, INT4) to shrink model size and accelerate inference with minimal accuracy loss.

— GGUF is a binary file format designed for storing quantized models for efficient CPU inference, widely used in llama.cpp.

— Mixture of Experts (MoE) activates only a subset of specialized feed-forward networks (experts) for each input token, keeping total parameters high while bounding compute cost.

— Mamba is a Selective State Space Model that replaces the attention mechanism with a data-dependent state space, offering linear-time sequence processing.

— Direct Preference Optimization (DPO) aligns a language model to human preferences directly from pairwise comparison data, without needing a separate reward model.

— Constitutional AI trains a model to self-critique and revise its own outputs against a written set of principles, reducing reliance on human feedback.

— Guided Generation constrains an LLM's output to a specified format (like JSON or a regular expression) by masking invalid tokens during decoding.

— Retrieval-Augmented Generation (RAG) fetches relevant documents from an external knowledge base and inserts them into the prompt, grounding the model's answer in retrieved evidence.

— Prompt Injection is a security attack where untrusted data in a prompt overrides the system's original instructions, causing the model to behave in unintended ways.

Conclusions

KV Cache is the single biggest memory consumer during LLM inference; every optimization from MQA to PagedAttention is fundamentally a strategy to shrink or better manage that cache.

Speculative decoding is a rare free lunch: it produces mathematically identical output to standard autoregressive decoding while cutting latency, because verification is much cheaper than generation.

The shift from RLHF to DPO and Constitutional AI reflects a broader push to simplify the alignment pipeline — removing the reward model reduces a whole class of training instability.

Mamba's linear-time complexity challenges the assumption that attention is irreplaceable, but its real-world adoption still lags behind the enormous ecosystem built around Transformer optimizations like FlashAttention.

Concepts & terms

KV Cache

A memory store that retains the Key and Value vectors from previous tokens during autoregressive decoding, so the model does not recompute them for each new token.

FlashAttention

An exact-attention algorithm that reorganizes computation to minimize reads and writes to GPU high-bandwidth memory, using tiling and kernel fusion to accelerate attention layers.

Speculative Decoding

An inference technique where a small draft model proposes multiple future tokens, and the large target model verifies them in one parallel forward pass, reducing latency without changing the output distribution.

Mixture of Experts (MoE)

An architecture where each input token is routed to only a small subset of available feed-forward sub-networks (experts), enabling models with trillions of parameters while keeping per-token compute roughly constant.

Mamba

A neural architecture based on Selective State Space Models that processes sequences with linear time complexity, avoiding the quadratic cost of self-attention.

Direct Preference Optimization (DPO)

A fine-tuning method that directly optimizes a policy from human preference pairs without training a separate reward model, simplifying the RLHF pipeline.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗