跪拜 Guibai
← All articles
Artificial Intelligence

Train Your Own 7.5M-Parameter GPT from Scratch: The Full Pipeline

By 日习一技 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Toy-scale training makes the entire LLM pipeline tangible enough to run on a laptop in minutes. It exposes the mechanics behind tokenization, optimization, and fine-tuning, and clarifies what a tiny model can and cannot learn.

Summary

A 7.5M-parameter GPT is trained from scratch on a 314KB toy corpus (8 sentence patterns repeated 4000 times). The pipeline follows a complete LLM training series: a BPE tokenizer is trained with Hugging Face tokenizers (configured vocab 2048, ends up with 428 actual tokens), a mini GPT is hand-written with token and positional embeddings, 4 transformer blocks, LayerNorm and an LM head. Training uses cross-entropy loss, AdamW, cosine annealing, gradient clipping, and runs 1000 steps. Memory peaks around 120 MiB; gradient accumulation demonstrates how to emulate larger batches with smaller memory. Loss drops from 6.23 (close to the theoretical random-guess ln(428)=6.06) to near zero within 100 steps, showing the model memorizes the corpus rather than generalizing. Generation turns from gibberish to verbatim recitation. Then SFT with 12 instruction-response pairs runs 300 steps at lr 1e-4, teaching the model to stop at an end token and answer questions. The entire pipeline is economically feasible on a laptop, and the exercise ties together every concept from a 11-part series on pretraining, SFT, DPO/GRPO, etc.

Takeaways
— The toy corpus is 8 sentence patterns repeated 4000 times, totaling 314 KB.
— BPE tokenizer configured with vocab_size 2048 actually learns only 428 tokens due to tiny corpus.
— The mini GPT has 7.48M parameters: hidden dim 384, 6 heads, 4 layers, context 128.
— Training runs 1000 steps with AdamW (lr 3e-4, weight_decay 0.1), cosine annealing to 0, gradient clipping at 1.0.
— Initial loss is 6.23, matching theoretical random cross-entropy ln(428) ≈ 6.06.
— Loss drops to 0.01 by step 100 and stays near zero, indicating memorization.
— Memory usage peaks around 120 MiB for the 7.5M model.
— Gradient accumulation with 4 micro-batches of batch_size 8 emulates a batch of 32, dividing loss by accumulation steps.
— SFT uses 12 instruction-response pairs, runs 300 steps at lr 1e-4, and adds a stop token to end generation.
— After SFT, the model answers questions and stops at <|endoftext|>.
— The whole process runs on a laptop with CPU possible; CUDA optional.
Conclusions

The initial loss sanity check against ln(vocab_size) is a cheap, powerful way to catch implementation errors before wasting compute.

The loss curve drops to near zero because the model is memorizing the eight-sentence corpus, not generalizing; the tutorial explicitly frames the exercise as a demonstration of memorization.

Skipping distributed training and RLHF at toy scale is a deliberate choice that keeps the code readable, while still mapping every step to a real-world counterpart.

The author's use of Feynman learning (teaching to expose gaps) and iterative writing over three months is a practical method for self-taught ML developers.

Concepts & terms
BPE (Byte-Pair Encoding)
A tokenization algorithm that iteratively merges the most frequent adjacent byte pairs to build a fixed-size vocabulary.
SFT (Supervised Fine-Tuning)
Training a pretrained model on instruction-response pairs to make it follow conversational formats.
Gradient Accumulation
A technique that aggregates gradients over several micro-batches before updating weights, simulating a larger batch size without increasing memory.
Perplexity
e^(cross-entropy loss), a measure of how surprised the model is by the next token; lower is better.
Source: juejin.cn ↗ Google Translate ↗ Backup ↗