Interview · Artificial Intelligence · Programmer

From a Single Neuron to Attention: A Straight-Line Path Through Deep Learning's Core Architectures

By 爱敲代码的小黄 · Nov 9, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Because each architecture appears as the fix for a specific failure, from vanishing gradients to the fixed-length bottleneck to sequential computation, the reason chain behind Transformer stays visible instead of dissolving into buzzwords. It works as a reference for interview prep and for engineers who know the model names but not why each one had to exist.

Summary

The tutorial builds from the smallest unit up: a neuron is a weighted sum plus a bias through an activation function, and without that activation a stack of layers still collapses into one linear transform. Training is the four-step loop of forward pass, loss, backpropagation, and optimizer updates, and the deep part comes from vanishing and exploding gradients, with ReLU, Xavier/He initialization, and batch normalization as mitigations that reduce but never guarantee the problem away.

CNN handles images through weight-shared convolution kernels and pooling that shrink feature maps while keeping key information; receptive fields widen as layers stack, from edges to entire objects, and ResNet's residual connections gave very deep networks a gradient highway. RNN handles order through a hidden state passed step by step, but vanilla RNNs can't parallelize and lose long-range dependencies, so LSTM added a cell state with forget, input, and output gates, GRU merged those gates, and BiLSTM reads both directions.

Encoder–Decoder splits understanding from generation, compressing the input into a fixed vector that becomes a bottleneck on long sentences, and Teacher Forcing hides the error accumulation that appears at inference. Attention fixes the bottleneck by letting the decoder re-scan every encoder position and weight the most relevant ones per step. That sets up Transformer, which removes recurrence entirely and makes attention the core of the model, allowing direct connections between any two tokens and parallel processing.

Takeaways

— A neuron computes a weighted sum of its inputs plus a bias, then applies an activation function; without the activation, endlessly stacked layers still collapse into one linear transformation.

— Training loops over four steps: forward propagation, loss computation, backpropagation via the chain rule, then optimizer parameter updates. Inference runs only forward propagation.

— ReLU, Xavier/He initialization, and batch normalization mitigate vanishing gradients, but the article is explicit that they cannot guarantee elimination.

— CNN convolution kernels are weight-shared across the whole image, so each kernel learns one pattern such as edges or direction and finds it anywhere; stacked layers widen the receptive field from pixels to full objects.

— Pooling (max pooling) or strided convolution shrinks feature maps and adds tolerance to small feature shifts; LeNet, AlexNet, VGG, then ResNet's residual connections enabled progressively deeper networks.

— RNNs transmit the hidden state h_t across time steps, not the output o_t, and the U, W, V matrices are shared at every time step.

— Vanilla RNNs can't parallelize because each step depends on the previous one, and long chains cause gradients to vanish or explode, which is why they struggle with long-range dependencies.

— LSTM adds a cell state with forget, input, and output gates that decide what to discard, write, and read; GRU merges those into update and reset gates with fewer parameters, and BiLSTM reads both directions but can't be used for left-to-right generation.

— The traditional Encoder–Decoder compresses the whole input into the encoder's last hidden state, and that fixed-length vector becomes a bottleneck for long sentences.

— Teacher Forcing trains the decoder on correct previous tokens, while inference feeds self-generated tokens, so an early error can accumulate.

— Attention has four steps: keep all encoder positions, score relevance against the decoder state, softmax-normalize into weights, and return a weighted context vector.

— Transformer's move is to drop recurrence and make attention the whole model, so any two tokens connect directly and the full sequence can be processed in parallel.

Conclusions

Every architecture appears as the remedy for the previous one's failure, from RNN's long-range loss to LSTM's gates, from the fixed-vector bottleneck to Attention, and from sequential computation to Transformer. That framing, more than any formula, is what makes the history cohere.

LSTM gets honest treatment: its gates improve long-term memory, but sequential, hard-to-parallelize computation survives unchanged, and only Transformer removes it. Tutorials often present LSTM as the endpoint of sequence modeling, so drawing that boundary is the useful distinction.

The article flags its own notation inconsistency, where s and h swap meaning between the diagram and the formulas. Attention literature is full of such symbol drift, and anyone comparing figures to equations has to be alert to it.

Teacher Forcing trains the decoder on correct previous tokens, but inference feeds self-generated ones, so a wrong first step compounds into longer errors. The piece makes that training-inference gap explicit instead of glossing over it.

Concepts & terms

Receptive field

The range of the original input a given neuron can indirectly see. It grows as convolutional layers stack, which is how deep CNNs assemble local edges into whole objects.

Weight sharing

Reusing the same convolution kernel across the entire image. It cuts parameter count and lets a single kernel detect a feature wherever it appears.

Residual connection

A skip path that lets input x bypass layers and merge later as F(x)+x, giving gradients and information a highway through very deep networks. It is ResNet's core idea.

Teacher Forcing

Training a decoder on the correct previous token rather than its own prediction. It stabilizes learning but hides the error accumulation that surfaces at inference, when the model must feed its own output.

Vanishing and exploding gradients

Backpropagated gradients that shrink to near zero over many layers, so early parameters stop learning, or get amplified so large they destabilize training. This is why deep networks need ReLU, careful initialization, and batch normalization.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗