---

theme: smartblue

If deep learning is a tree, then the neural network is the trunk, CNN and RNN are two important branches, Encoder–Decoder separates "understanding input" from "generating output", and Attention is the key step toward Transformer.

This article doesn't aim to list terms, but follows one main thread: what happens to data after it enters a model, and how the model learns.

一、神经网络究竟在做什么？

A neural network is essentially a function that can automatically adjust its parameters through data. Input can be a set of numbers, an image, or a sentence; output can be a score, an image category, or another sentence.

A typical neural network has three parts:

• Input layer: receives raw data;
• Hidden layer: continuously extracts and combines features;
• Output layer: gives the final result.

A model with only input and output layers is usually called a single-layer network; one with a single hidden layer is commonly called a shallow neural network; one with multiple hidden layers is what we call a deep neural network.

1. 神经元：最小的计算单元

A neuron can be understood as a small calculator. It first assigns a weight to each input, adds a bias, and finally passes through an activation function:

$$ y=f(w_1x_1+w_2x_2+\cdots+w_nx_n+b) $$

where:

• $x$ is the input;
• $w$ is the weight, indicating how important each input is;
• $b$ is the bias, letting the model shift the decision boundary as a whole;
• $f$ is the activation function, adding nonlinearity to the network.

Without an activation function, no matter how many layers are stacked, the whole network is ultimately equivalent to a single linear transformation and cannot learn complex patterns. ReLU is one of the most commonly used activation functions: when input is greater than 0 it outputs as-is, when less than 0 it outputs 0. It is simple to compute and can also mitigate vanishing gradients to some extent.

2. 模型是怎样学会的？

Using "predicting exam score based on study time" as an example, the simplest model is:

$$ \hat{y}=wX+b $$

At the start of training, $w$ and $b$ are usually randomly initialized, so predictions are quite inaccurate. Then the model repeatedly performs four steps:

1. Forward propagation: input study time, compute predicted score layer by layer;
2. Compute loss: compare predicted score with true score to measure how wrong it is;
3. Backpropagation: use the chain rule to compute each parameter's influence on the loss, i.e., the gradient;
4. Update parameters: the optimizer adjusts weights and biases in the direction that decreases the loss.

This can be analogized to a student doing exercises: first write the answer, then check the answer, analyze which step the error came from, then adjust the approach. After many rounds of training, the loss gradually decreases, and the parameters "remember" the patterns in the data.

After training, the model enters the inference stage. At this point it only uses the learned parameters for forward propagation; it no longer computes loss, backpropagation, or parameter updates.

3. 为什么深层网络不好训练？

A gradient can be understood as a "correction suggestion" the model gets from its mistakes. During backpropagation, this suggestion passes from the last layer forward layer by layer. If it shrinks at every layer, by the time it reaches the front it may be nearly 0, making it hard for the front parameters to keep learning; this is vanishing gradient. If the gradient gets amplified layer by layer, it can form exploding gradient.

Three common mitigation methods:

• Use ReLU activation: it acts like a gate, letting positive values pass directly, so gradients pass more easily in the positive region;
• Reasonable parameter initialization: Xavier and He initialization are like adjusting the initial volume on an audio system, keeping weights neither too large nor too small;
• Use batch normalization: Batch Normalization (BN) acts like a "coordinator" that adjusts each layer's output to a relatively stable range, making training smoother.

These three methods respectively improve the activation mode, the training starting point, and the stability of intermediate data; they only mitigate vanishing gradients and cannot guarantee it disappears entirely. They also don't each optimize a single parameter; the network's parameters are ultimately computed by backpropagation for gradients and updated together by the optimizer.

二、CNN：怎样让神经网络看懂图片？

To a computer, an image is just a pixel matrix. If all pixels were directly connected to a fully connected layer, the number of parameters would be enormous, and it would also ignore the most important characteristics of images: adjacent pixels are closely related, and the same feature can appear anywhere.

Convolutional Neural Network (CNN) is designed precisely for these two characteristics.

1. 卷积层：用小窗口寻找局部特征

A convolution kernel is a small weight matrix, commonly sized $3\times3$. It is like a sliding small window that scans the whole image from left to right and top to bottom. At each position, it element-wise multiplies with the covered region and sums to produce a new value.

The same convolution kernel is reused across the entire image, so:

• There are fewer parameters; this is called weight sharing;
• No matter where a feature appears in the image, the same kernel can recognize it;
• Each convolution kernel learns one pattern, such as edges, orientation, or texture.

Color images usually have three channels: red, green, blue. The convolution kernel processes all three channels simultaneously. After one kernel scans, it produces a feature map; 16 kernels produce 16 feature maps, each recording a different feature discovered by the model.

pixels → edges and colors → textures and shapes → local structures like eyes, tires → whole objects

The first convolutional layer can only see a small local region; as convolutional layers stack, later neurons can indirectly see a larger region of the image. This range is called the receptive field. The growing receptive field is an important reason deep CNNs can progress from local edges to understanding whole objects.

Convolution also has three common parameters:

• Kernel size determines the observation range
• Stride determines how far it moves each time
• Padding determines whether to pad values at edges to control output size.

2. 池化层：保留重点，压缩尺寸

A pooling layer summarizes a small region. For example, max pooling keeps only the maximum value in the region. It shrinks feature maps, reduces computation, and makes the model less sensitive to small shifts in features.

Modern CNNs sometimes replace pooling with strided convolution, but the goal is similar: reduce spatial size while preserving key information as much as possible.

3. 一张图片如何完成分类？

A typical data flow is:

input image → multiple rounds of convolution and activation → downsampling → high-level features → summarizing features → classification layer outputs probabilities for each class

For example, when recognizing a cat, the front layers may detect edges, middle layers combine into ears and eyes, and deep layers combine these features into "cat". During training, classification error is still backpropagated through all convolution kernels, and the kernels gradually learn to extract useful features.

After convolution, the model flattens or summarizes the multiple feature maps and hands them to the fully connected layer. With two fully connected layers, the first combines features like ears, whiskers, and contours; the second computes raw scores (logits) for classes like cat and dog. Softmax then converts these scores into probabilities, and the model picks the class with the highest probability.

feature maps → flatten or summarize → combine features → class scoring → Softmax probabilities → prediction result

The number of fully connected layers isn't fixed. Classic CNNs often use multiple fully connected layers; modern CNNs often use global average pooling plus one fully connected layer to reduce the parameter count.

4. CNN 为什么越做越深？

LeNet validated the effectiveness of convolutional structures; AlexNet truly drew attention to deep CNNs; VGG built deeper networks with successive small kernels. But networks aren't necessarily better when deeper: as layers increase, gradient and information propagation both become difficult, and "deeper but worse" degradation can even appear.

ResNet's key is the residual connection:

$$ y=F(x)+x $$

It lets input $x$ bypass several layers and pass directly to the later part, essentially building a highway for information and gradients, making very deep networks easier to train. Later DenseNet further establishes denser connections between different layers to promote feature reuse.

三、RNN：怎样让神经网络理解顺序？

Images are fundamentally about spatial relationships, while text, speech, and time series are fundamentally about order. "I hit him" and "He hit me" contain the same characters but mean completely different things.

Recurrent Neural Network (RNN) reads data in order and uses a hidden state to preserve the information it has seen.

1. 文字如何进入 RNN？

RNNs can't process text directly; text must first be converted to numeric vectors:

text → tokens → token IDs → embedding vectors → RNN

For example, "我喜欢学习" is first segmented into "我", "喜欢", "学习", then converted into corresponding IDs according to the vocabulary. Since IDs are just numbers, the tokens pass through an embedding layer that converts each token into a vector carrying semantic information. In RNN diagrams, $x_t$ is the vector for the $t$-th token.

Training and inference must use the same tokenization rules, vocabulary, and embedding parameters; otherwise the model can't correctly understand the input.

2. RNN 如何传递上下文？

At the $t$-th time step, the RNN receives the current token's vector $x_t$ and the previous step's memory $h_{t-1}$:

$$ h_t=f(Ux_t+Wh_{t-1}+b) $$

$$ o_t=g(Vh_t+c) $$

The new hidden state $h_t$ goes in two directions: on one side it generates the current output $o_t$ through $V$, and on the other side it is passed to the next moment as context. In other words, what is transmitted horizontally is $h_t$, not $o_t$. Only in tasks like text generation could the predicted token also become the input for the next moment.

The three matrices in the diagram each have their role:

• $U$: converts the current input $x_t$ into the hidden space;
• $W$: passes the previous moment's memory $h_{t-1}$ to the current moment;
• $V$: converts the current hidden state $h_t$ into output $o_t$.

These parameters are shared across all time steps. That means RNN doesn't build a new network for each word in a sentence; the same unit is reused repeatedly.

Take "我喜欢自然语言处理" (I like natural language processing): when the model reads "我", it forms a memory; when it reads "喜欢", it combines the current word with the previous memory; as it keeps reading, the hidden state continuously updates. The final state contains the context the model considers important.

3. 普通 RNN 的两个难题

First, each time step depends on the previous one, so training is hard to parallelize the way CNN can be.

Second, early information has to travel through a long chain. During forward propagation it may be diluted by later information; during backpropagation, gradients go through many multiplications and easily vanish or explode. This makes it hard for vanilla RNNs to learn long-range dependencies.

4. LSTM：给记忆增加三道门

LSTM (Long Short-Term Memory) mainly mitigates vanilla RNN's long-term information loss and vanishing gradient, but it doesn't solve sequential computation or the difficulty of parallelization. Beyond the hidden state, it adds a more stable "cell state" and uses gating mechanisms to decide how information flows.

The three gates can be understood like this:

• Forget gate: how much of the old memory to keep, which content to drop;
• Input gate: which new information in the current input is worth writing into long-term memory;
• Output gate: which content from memory to take out at the current moment as output.

Gate values are typically between 0 and 1; close to 0 means "block", close to 1 means "let through". These gates are not hand-written rules; they are learned from training data. The cell state provides a smoother information channel, so LSTM preserves long-term information better than vanilla RNN.

For example, when reading "中国", the input gate can write relevant information into memory; while processing the long middle content, the forget gate decides whether to keep it; when reading "母语", the output gate retrieves this memory to help the model predict "中文".

5. GRU 与 BiLSTM

GRU is a simplified version of LSTM. It merges multiple states and gates, mainly using an update gate and a reset gate. It has fewer parameters, trains faster, and its practical performance is often close to LSTM's.

BiLSTM uses two LSTMs to read the sequence from left to right and from right to left respectively, then merges both sides' information. This way, understanding a word can see both the preceding and following context, which suits tasks like text classification and named entity recognition. But when generating text word by word, future words don't exist yet, so full bidirectional information can't be relied on directly.

四、Encoder–Decoder：为什么要把“理解”和“生成”分开？

If the task only requires judging the input, a feature extraction module plus a classifier usually suffices; if the task requires first understanding the input and then generating a new sequence, Encoder–Decoder is appropriate. It is the general framework for "sequence-to-sequence" tasks, commonly used in machine translation, summarization, dialogue, and speech recognition. Input and output lengths can differ, and content and form don't need to correspond one-to-one.

It runs in two phases.

1. Encoder：把输入读懂

The encoder reads each word of the source sentence in turn, continuously fusing the word vector with the existing hidden state:

input sentence → word vectors → progressively updating hidden state → semantic representation of the input

A traditional RNN Encoder–Decoder usually treats the encoder's last hidden state as the compressed representation of the whole sentence, then hands it to the decoder.

2. Decoder：根据语义逐步生成

The decoder receives the encoding result and starts generating from the start token <BOS> word by word:

1. Based on the encoding result and <BOS>, predict the first word;
2. Feed the just-generated word as the next input, predict the second word;
3. Repeat until the end token <EOS> is generated.

At each generation step, the decoder computes raw scores for all tokens in the target vocabulary based on its current state, converts them to probabilities via Softmax, and selects one token:

Decoder state → scores for each token in vocabulary → Softmax probabilities → output token → as next step's input

During training, the correct answer from the previous step is often fed to the decoder to help it learn stably; this is called Teacher Forcing. At inference there is no correct answer, so it can only use the words it just generated itself; if an early error occurs, errors can continue to accumulate.

3. 固定长度向量为什么不够？

The traditional structure requires the encoder to compress the whole sentence into a fixed-length vector. The longer the sentence, the more this vector becomes an information bottleneck: early details may be lost in transmission, and the decoder has no way of knowing which part of the original sentence to focus on when generating the current word.

Attention was introduced precisely to solve this problem.

五、Attention：需要什么，就回头看什么

In the classic structure without Attention, the Encoder hands its final state $h_n$ to the Decoder. The Decoder then, combining the previous token, continuously updates its own states $s_1、s_2……$. These states record the generation process; they don't mean it can look back at the Encoder's $h_1、h_2……$. What Attention adds is exactly this access channel.

The core idea of Attention (attention mechanism) is very intuitive: each time the decoder generates a word, it re-examines all the encoder's outputs and picks the information most relevant at that moment.

Suppose we translate "我喜欢机器学习" (I like machine learning) into English. When the decoder is about to generate "machine", it should focus more on "机器"; when generating "learning", it should focus more on "学习", rather than always relying on the same compressed sentence vector.

One attention computation can be divided into four steps:

1. Save all information: the encoder doesn't just keep the last state, but retains a representation of each input position;
2. Compute relevance: match the decoder's current state against each encoder state to get a set of scores;
3. Normalize weights: use Softmax to turn scores into attention weights summing to 1;
4. Weighted summarization: positions with higher weights contribute more, finally yielding a context vector specific to the current step.

In the diagram green represents Encoder states and pink represents Decoder states; the diagram uses $s$ for the Encoder and $h$ for the Decoder, opposite to the symbol conventions in the text. Attention weights indicate the degree of attention to each position in the source text; they are not output probabilities over the English vocabulary. The decoder combines the weighted-summed information with its own state, then predicts the next token.

Can be abbreviated as:

$$ \alpha_{t,i}=\operatorname{softmax}(\operatorname{score}(s_{t-1},h_i)) $$

$$ c_t=\sum_i \alpha_{t,i}h_i $$

$h_i$ is the encoder's representation at position $i$, $s_{t-1}$ is the decoder's current state used for querying, $\alpha_{t,i}$ is the degree of attention to that position, and $c_t$ is the context needed for this generation step.

So the original flow:

entire input → one fixed vector → entire output

becomes:

entire input → retain all positions → dynamically select relevant information at each generation step

Attention mitigates the information bottleneck of long sequences and also makes the model's alignment easier to observe. For example, when translating a particular word, we can see which words in the source sentence the model mainly attended to.

六、Attention 之后，为什么还需要 Transformer？

Attention mitigates the information bottleneck of the traditional Encoder–Decoder: the decoder no longer relies only on a fixed-length vector, but can re-find the most relevant information in the input when generating each token.

But at this point the main body is still RNN. Tokens still have to be processed one by one in order; each time step depends on the previous one, so parallelization is hard; and two tokens far apart still need to pass through a long information transmission path.

So a new problem emerged:

Can we remove the recurrent structure, let any two tokens establish a direct connection, and process the entire sequence simultaneously?

Transformer's answer: turn Attention from an auxiliary mechanism into the core of the whole model. As for how it uses Self-Attention, QKV, positional encoding, and multi-head attention, that will be broken down in the next article.

Looking back along this article's path, the technical evolution is actually very natural:

RNN relies on hidden states to pass context → LSTM improves long-term memory → Encoder–Decoder separates understanding from generation → Attention dynamically finds relevant information → Transformer makes Attention the core

结语

Neural network learning is essentially a "predict–compare–correct" loop; CNN understands images through local connections and weight sharing; RNN passes context through hidden states; LSTM and GRU manage memory with gating mechanisms; Encoder–Decoder splits understanding and generation into two phases; Attention lets the model actively find the most relevant information at every step.

Here we've completed the full route from an ordinary neural network to Attention. The next article will formally enter Transformer, seeing how it escapes RNN's sequential computation and how it lets every token directly find information related to itself.