Train Your Own 7.5M-Parameter GPT from Scratch: The Full Pipeline
Toy-scale training makes the entire LLM pipeline tangible enough to run on a laptop in minutes. It exposes the mechanics behind tokenization, optimization, and fine-tuning, and clarifies what a tiny model can and cannot learn.
A 7.5M-parameter GPT is trained from scratch on a 314KB toy corpus (8 sentence patterns repeated 4000 times). The pipeline follows a complete LLM training series: a BPE tokenizer is trained with Hugging Face tokenizers (configured vocab 2048, ends up with 428 actual tokens), a mini GPT is hand-written with token and positional embeddings, 4 transformer blocks, LayerNorm and an LM head. Training uses cross-entropy loss, AdamW, cosine annealing, gradient clipping, and runs 1000 steps. Memory peaks around 120 MiB; gradient accumulation demonstrates how to emulate larger batches with smaller memory. Loss drops from 6.23 (close to the theoretical random-guess ln(428)=6.06) to near zero within 100 steps, showing the model memorizes the corpus rather than generalizing. Generation turns from gibberish to verbatim recitation. Then SFT with 12 instruction-response pairs runs 300 steps at lr 1e-4, teaching the model to stop at an end token and answer questions. The entire pipeline is economically feasible on a laptop, and the exercise ties together every concept from a 11-part series on pretraining, SFT, DPO/GRPO, etc.
The initial loss sanity check against ln(vocab_size) is a cheap, powerful way to catch implementation errors before wasting compute.
The loss curve drops to near zero because the model is memorizing the eight-sentence corpus, not generalizing; the tutorial explicitly frames the exercise as a demonstration of memorization.
Skipping distributed training and RLHF at toy scale is a deliberate choice that keeps the code readable, while still mapping every step to a real-world counterpart.
The author's use of Feynman learning (teaching to expose gaps) and iterative writing over three months is a practical method for self-taught ML developers.