Train Your Own 7.5M-Parameter GPT from Scratch: The Full Pipeline
LLM Training Series Finale: Train Your Own Tiny Model from Scratch
Author: 日习一技
In the previous article, we learned two routes for preference alignment: DPO bypasses the reward model and reinforcement learning, directly using preference data for supervised training; GRPO replaces the value network with group-relative advantages, making reinforcement learning lighter. At this point, we've examined the entire pipeline of LLM training: data cleaning, tokenizer training, objective functions, backpropagation and optimizers, memory and mixed precision, distributed parallelism, pretraining engineering, SFT, RLHF, and DPO.
However, just reading without practice is not enough. Today is the final part of the series; we won't introduce new concepts. Instead, we'll get our hands dirty and train a small model from scratch with a few hundred lines of code, running through every stage mentioned in the previous eleven articles. After this, you'll see the loss drop dramatically from near random guessing, the model's output turn from gibberish to whole passages of the corpus, and after feeding it a dozen instruction examples, it can answer your questions.
Benchmark Projects
Training a model from scratch is nothing new; there are several well-known benchmarks in the community:
- Karpathy's series of projects: nanoGPT claims to replicate GPT training with minimal code; the core is only
model.pyandtrain.py, and on the tiny Shakespeare dataset it can train a character-level GPT that writes Shakespearean dialogue. build-nanogpt hands-on replicates the full GPT-2 training process. nanochat, released in October 2025, goes further by putting the tokenizer, pretraining, SFT, and GRPO reinforcement learning pipeline into about 8,000 lines of code; with 8 H100s running for 4 hours and costing about $100, you can train a 561M-parameter chat model—almost the same roadmap as ours, just scaled up by several orders of magnitude. - minimind: This repository in the Chinese community lowers the barrier to entry, claiming you can train a tens-of-millions-parameter model with simple chat capabilities from scratch using an ordinary GPU in a few hours, with very complete accompanying tutorials.
- Happy-LLM: A systematic tutorial by Datawhale, starting from Transformer principles, hand-writing LLaMA2, and going through pretraining, SFT, and GRPO.
- train-llm-from-scratch: The most comprehensive personal project, covering everything from raw text to pretraining, SFT, reward models, PPO, DPO, and GRPO, all hand-written without relying on
transformersandtrl. - LLMs-from-scratch: The companion code to Sebastian Raschka's "Build a Large Language Model from Scratch", progressing chapter by chapter from tokenization to instruction fine-tuning, with a Chinese translation available; suitable for those who like following a book.
This article borrows ideas from these projects, but all code is written by ourselves, aiming for simplicity. Our goal is not to train a strong model, but to let every concept run in our hands.
Overall Roadmap
The complete roadmap is shown below; each step corresponds to a previous article:
We can see that this roadmap is a microcosm of the entire series. Distributed parallelism and RLHF are not used at this toy scale, so we won't force a demonstration.
Step 1: Prepare a Toy Corpus
The first step is data. As discussed in Article 2, real LLM pretraining uses TB-scale internet text, requiring extensive cleaning, deduplication, and mixing ratios. Our model is small, so a small toy corpus suffices.
We use small data for two reasons. First, our goal is not model capability but to run the training pipeline; a small corpus can show visible results in minutes. Second, a small model with large data leads to underfitting—the model capacity cannot absorb the patterns in the data; conversely, a small model with small data makes the loss drop nicely and generation effects more intuitive. This also aligns with scaling law intuition.
We prepare a small mixed Chinese-English text, saved as toy_corpus.txt, centered around a fictional character with repeated sentence structures to help the model learn patterns quickly:
小明是一个喜欢编程的少年。小明每天早上都会写代码。
Xiaoming is a boy who loves programming. He writes code every morning.
编程让小明感到快乐。Programming makes Xiaoming happy.
大模型是一种会说话的程序。A large language model is a program that can talk.
训练一个小模型需要数据、算力和耐心。Training a small model needs data, compute and patience.
...
In the actual demo, we replicate this corpus 4000 times, totaling 8 sentence patterns, with a size of 314 KB.
If you want to be closer to reality, you could also use an open-source small corpus, such as the tiny Shakespeare dataset from nanoGPT. The idea is exactly the same, just a different text style.
Step 2: Train a BPE Tokenizer
With the corpus ready, we turn it into tokens. From Article 3, we learned that LLMs don't directly process characters; they first split text into token sequences. The most common splitting algorithm is BPE (Byte-Pair Encoding), which starts from bytes and repeatedly merges the most frequent adjacent pairs to learn a fixed-size vocabulary.
Real models have tokenizers with vocabularies of hundreds of thousands, e.g., GPT-3.5 and GPT-4 use cl100k_base encoding with a vocabulary just over 100k (explicitly recorded in OpenAI's official tokenizer library tiktoken). Our corpus and model are small, so there's no need for a large vocabulary; we set vocab_size=2048. Training our own tokenizer with Hugging Face's tokenizers library takes only a few lines of code:
from tokenizers import Tokenizer, models, trainers, pre_tokenizers
# Start with byte-level BPE; any character (including Chinese and emoji) can be split into bytes first, avoiding unknown tokens
tokenizer = Tokenizer(models.BPE())
# ByteLevel pre-tokenizer converts text to byte sequences
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)
# Vocabulary size 2048 is an upper limit; small corpus may not fill it
# Three special tokens: end-of-text, and reserved for SFT conversation role markers
trainer = trainers.BpeTrainer(
vocab_size=2048,
special_tokens=["<|endoftext|>", "<|user|>", "<|bot|>"],
)
# Train the tokenizer on the corpus file
tokenizer.train(["toy_corpus.txt"], trainer)
# Save the trained tokenizer as JSON for later loading
tokenizer.save("mini_tokenizer.json")
As you see, besides normal text tokens, we register three special tokens: <|endoftext|> marks the end of text, <|user|> and <|bot|> are reserved for the SFT stage to mark conversation roles. After training, verify:
from tokenizers import Tokenizer, decoders
# Load the trained tokenizer
tokenizer = Tokenizer.from_file("mini_tokenizer.json")
# Byte-level BPE needs a matching decoder, otherwise Chinese decodes as gibberish
tokenizer.decoder = decoders.ByteLevel()
# Encode: text becomes token IDs
ids = tokenizer.encode("小明喜欢编程").ids
print(ids)
# Decode: token IDs back to text
print(tokenizer.decode(ids))
Output:
[115, 253, 305]
小明喜欢编程
One sentence is split into 3 tokens, decoding back exactly.
Note: Although the configured vocabulary is 2048, the toy corpus has only 8 sentence patterns; BPE merges all possible pairs, and in the end only 428 tokens are learned. The actual vocabulary size is 428. We'll use this real vocabulary size for parameter count and initial loss later.
With that, the bridge between text and tensors is built. We encode the whole corpus into a long token sequence, totaling 46,000 tokens, split into training and validation sets at a 9:1 ratio, saved as train_ids and val_ids.
Step 3: Write a Mini GPT by Hand
This is the core step. Article 4 covered the language model objective of predicting the next token, and Article 5 covered backpropagation and optimizers. Now we put them into a real model.
Our mini GPT has a textbook structure: Token Embedding plus Positional Embedding, followed by 4 Transformer Blocks, and finally a LayerNorm and a Language Model (LM) Head that outputs a probability distribution over the vocabulary. Each Block contains causal-masked multi-head self-attention and a feed-forward network, both parts from Article 4.
Note: We use the GPT-2 style learnable positional embedding, not sinusoidal or RoPE. It's essentially a lookup table where each position stores a vector, trained along with the model. We choose it for simplicity—two lines of code.
The code, simplified to keep only the main parts:
import torch
import torch.nn as nn
import torch.nn.functional as F
# A Transformer Block: self-attention + feed-forward, each with a residual connection
class Block(nn.Module):
def __init__(self, d, n_head, max_len):
super().__init__()
# Multi-head self-attention, batch_first=True means input shape [batch, seq_len, dim]
self.attn = nn.MultiheadAttention(d, n_head, batch_first=True)
# Feed-forward: expand dims by 4x, GELU activation, then project back to d
# Expand-and-compress is a Transformer staple, giving the model a larger intermediate space for nonlinear transformations
self.ffn = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
# Two LayerNorms, before attention and feed-forward
self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
# Causal mask: an upper triangular matrix; True positions block attention
# Position t can only see the first t tokens, not the future; this is unidirectional attention
mask = torch.triu(torch.ones(max_len, max_len), diagonal=1).bool()
# The mask is a constant and doesn't train, but it must be saved and moved with the model
# register_buffer registers it as a non-training state; using an ordinary attribute wouldn't follow the model
self.register_buffer("mask", mask)
def forward(self, x):
# x shape: [batch, seq_len, d], dim 1 is seq_len
# Actual seq_len may be less than max_len, so slice the mask accordingly
T = x.size(1)
# Pre-Norm: LayerNorm then attention (more stable training)
h = self.ln1(x)
# Three h's are query, key, value; self-attention is attention to itself
# Add residual connection: output = input + attention result, allowing gradient shortcuts
x = x + self.attn(h, h, h, attn_mask=self.mask[:T, :T])[0]
# Feed-forward also with LayerNorm and residual
return x + self.ffn(self.ln2(x))
class MiniGPT(nn.Module):
def __init__(self, vocab, d=384, n_head=6, n_layer=4, max_len=128):
super().__init__()
self.max_len = max_len
# Token embedding: lookup token IDs to d-dim vectors, vocab_size is the actual size
self.tok_emb = nn.Embedding(vocab, d)
# Positional embedding: lookup position indices (0, 1, 2, ...) to d-dim vectors
self.pos_emb = nn.Embedding(max_len, d)
# Stack n_layer Transformer Blocks
self.blocks = nn.Sequential(*[Block(d, n_head, max_len) for _ in range(n_layer)])
# Final LayerNorm
self.ln_f = nn.LayerNorm(d)
# LM head: project d-dim vectors to vocab-dim logits, one distribution per position
self.head = nn.Linear(d, vocab, bias=False)
def forward(self, idx, target=None):
T = idx.size(1)
# Add token and position vectors to get the first layer's input
x = self.tok_emb(idx) + self.pos_emb(torch.arange(T, device=idx.device))
# Through all blocks, LayerNorm, and LM head
# logits shape: [batch, T, vocab], raw scores for a probability distribution at each position
logits = self.head(self.ln_f(self.blocks(x)))
loss = None
# If target is provided, compute cross-entropy loss; otherwise only return logits (for generation)
if target is not None:
loss = F.cross_entropy(logits.view(-1, logits.size(-1)), target.view(-1))
return logits, loss
The configuration: hidden dim 384, 6 heads, 4 layers, context length 128. Print parameter count to verify:
# vocab uses the real vocabulary size
model = MiniGPT(vocab=tokenizer.get_vocab_size())
# numel is the number of elements in each parameter tensor; sum and divide by 1e6 for millions
print(sum(p.numel() for p in model.parameters()) / 1e6, "M")
7.48 M
About 7.5M parameters. For comparison, GPT-2 smallest is 124M, GPT-3 is 175B; our model is only a fraction of those but structurally identical, with every component present. The overall structure is shown below:
Next is the training loop, using everything from Article 5: Cross-Entropy Loss measures the gap between the predicted distribution and the true next token; AdamW is the standard optimizer for Transformers, decoupling weight decay from gradient updates; Cosine Annealing smoothly decreases the learning rate along a cosine curve; Gradient Clipping keeps the gradient norm within 1.0 to prevent exploding gradients.
import math
# Use GPU if available, otherwise CPU
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
# AdamW optimizer; lr is learning rate, weight_decay is weight decay (a regularizer)
opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.1)
# Cosine annealing scheduler: learning rate follows a cosine curve from 3e-4 to 0, T_max is total steps
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=1000)
# Sample random segments of token sequence; each batch has batch_size segments, each block_size tokens long
def get_batch(ids, batch_size=32, block_size=128):
ix = torch.randint(len(ids) - block_size - 1, (batch_size,))
# Input: tokens from i to i+128
x = torch.stack([ids[i:i + block_size] for i in ix])
# Target: input shifted right by one, i.e., the next token at each position
y = torch.stack([ids[i + 1:i + block_size + 1] for i in ix])
return x.to(device), y.to(device)
for step in range(1000):
x, y = get_batch(train_ids)
# Forward pass, compute cross-entropy loss
logits, loss = model(x, y)
# Clear previous gradients, then backpropagate to compute current gradients
opt.zero_grad()
loss.backward()
# Gradient clipping: if gradient norm exceeds 1.0, scale down to prevent exploding batches
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
# Update parameters, then step the scheduler
opt.step()
sched.step()
# Print every 50 steps and at the last step
if step % 50 == 0 or step == 999:
print(f"step {step:4d} | loss {loss.item():.2f} | lr {sched.get_last_lr()[0]:.2e}")
Note how get_batch constructs input and target: the target is the input shifted right by one—the way next-token prediction is trained. The model at position t sees the first t tokens and must guess the t+1-th.
Step 4: Observe Memory, Then Apply Gradient Accumulation
Before training runs, let's look at the protagonist of Article 6: memory. Insert two lines in the training loop:
# Print memory peak every 100 steps (for CUDA GPUs)
if step % 100 == 0:
used = torch.cuda.max_memory_allocated() / 1024 ** 2
print(f"step {step} | memory peak {used:.0f} MiB")
For the 7.5M model, it's only around 120 MiB; any GPU or even CPU is more than enough. But memory still holds quite a bit: weights, gradients, and two AdamW momentum states, plus activations that grow with batch and sequence length. If curious, you can increase batch_size and block_size a few times and watch the peak memory rise proportionally, matching the formulas from Article 6.
If batch size increases and memory becomes tight, the solution is Gradient Accumulation: instead of updating parameters every step, split a large batch into several micro-batches, run them sequentially, accumulate gradients for several steps, then update once. This emulates a large batch while keeping memory at the micro-batch level.
# Update once every 4 micro-batches, effectively enlarging batch by 4x
ACCUM = 4
for step in range(1000):
# Clear gradients at the start of each step; inner 4 micro-batches accumulate gradients
opt.zero_grad()
for micro in range(ACCUM):
# Micro-batch size is 8, keeping memory at small-batch levels
x, y = get_batch(train_ids, batch_size=8)
logits, loss = model(x, y)
# Divide loss by accumulation steps before backprop, keeping gradient scale consistent with a large batch
(loss / ACCUM).backward()
# After 4 micro-steps, clip, update, and move to next round
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
opt.step()
sched.step()
The key points: divide loss by accumulation steps before backprop; otherwise the effective learning rate scales up. Put zero_grad at the start of each outer step, so gradients accumulate across the 4 micro-batches and reset after each update.
Modified training loop:
Step 5: Watch Loss Drop and the Model Start Talking
Everything is ready; start training. The training log:
step 0 | loss 6.23 | lr 3.00e-04
step 50 | loss 0.31 | lr 2.98e-04
step 100 | loss 0.01 | lr 2.93e-04
step 200 | loss 0.01 | lr 2.71e-04
step 300 | loss 0.01 | lr 2.38e-04
step 400 | loss 0.01 | lr 1.96e-04
step 500 | loss 0.00 | lr 1.50e-04
step 600 | loss 0.01 | lr 1.03e-04
step 700 | loss 0.01 | lr 6.15e-05
step 800 | loss 0.01 | lr 2.84e-05
step 900 | loss 0.01 | lr 7.20e-06
step 999 | loss 0.01 | lr 0.00e+00
Here we do a sanity check: verify the implementation with a theoretically guaranteed value. No code needed; compare the loss at step 0 with the theoretical value. When the model hasn't learned, it's essentially guessing randomly. The actual vocab size is 428, so the cross-entropy of random guessing is ln(428) ≈ 6.06. The initial loss in the log is 6.23, very close, indicating the implementation is correct. If it deviated significantly, say 12 or 0.5, something is likely wrong.
Reading further, loss drops to 0.31 by step 50, then near zero by step 100. From the perplexity perspective, which is e^(cross-entropy), e^0.01 ≈ 1.01, meaning the model is almost certain about the next token. Plotting the loss curve shows this cliff-like drop:
We'll see later that this isn't the model learning general capabilities; it's memorizing the entire 8-sentence corpus. Training on a toy corpus is essentially a memorization demonstration.
Loss is just numbers; the real acceptance test is getting the model to talk. Write a simple generation function that samples the next token from the probability distribution repeatedly:
# no_grad disables gradient computation; generation doesn't need backprop, saving memory and speeding up
@torch.no_grad()
def generate(idx, max_new=40, temperature=0.8):
for _ in range(max_new):
# Sequence may exceed context length; take only the last max_len tokens
logits, _ = model(idx[:, -model.max_len:])
# Take the distribution at the last position, divide by temperature, then softmax to get probabilities
probs = F.softmax(logits[:, -1, :] / temperature, dim=-1)
# Sample the next token based on probabilities, append to sequence
idx = torch.cat([idx, torch.multinomial(probs, 1)], dim=1)
return idx
Here temperature is the temperature coefficient, controlling sampling randomness; values less than 1 make the distribution sharper and output more conservative. This is the same sampling parameter from the "LLM Inference" series. Give the model a start and compare outputs before and after training:
# Before training (step 0)
Input: 小明喜欢
Output: 小明喜欢小明每ery到快乐 mode loiteaner但 textw的箕 isgu代码orning……
# After training (step 1000)
Input: 小明喜欢
Output: 小明喜欢。The small model can talk.
小明是一个喜欢编程的少年。小明每天早上都会写代码。
Xiaoming is a boy who loves programming. He writes code every morning.
编程让小明感到快乐。Programming makes Xiaoming happy.……
Before training, the output is pure gibberish—special bytes, English letters, and Chinese characters randomly mixed. After 1000 steps, the output is fluent sentences, verbatim from the corpus, one after another. The model doesn't really understand who Xiaoming is; it has memorized the probability patterns of the text. This mutually confirms the loss dropping to zero.
Step 6: SFT, Making It Answer Questions
At this point, the model can only continue text, not converse. If you ask it a question, it treats the question as a continuation and keeps generating. In Article 9, we learned that the bridge from a continuation model to a dialogue model is SFT (Supervised Fine-Tuning): continue training on paired instruction-response data so the model learns to respond in a conversational format.
First, construct instruction data with a format different from pretraining text, using the role tokens we registered in the tokenizer:
sft_data = [
"<|user|>你是谁?<|bot|>我是小迷你,一个刚训练好的小模型。<|endoftext|>",
"<|user|>小明喜欢什么?<|bot|>小明喜欢编程,他每天早上都会写代码。<|endoftext|>",
"<|user|>训练模型需要什么?<|bot|>需要数据、算力和耐心。<|endoftext|>",
# ... a dozen more similar Q&A pairs, twelve in total
]
Training is almost the same as pretraining: still next-token prediction, still cross-entropy, but the data is in conversation format and the learning rate is smaller (1e-4). Note that the trailing <|endoftext|> is crucial—it's the signal for the model to stop. We run 300 steps, and the loss drops from about 8.5 to around 0.2:
# Smaller learning rate for SFT, taking smaller steps to avoid destroying pretrained language patterns
opt = torch.optim.AdamW(model.parameters(), lr=1e-4, weight_decay=0.1)
# Loop through 12 data items for 300 steps, meaning each Q&A is repeatedly trained about 25 times
for step in range(300):
text = sft_data[step % len(sft_data)]
ids = tokenizer.encode(text).ids
if len(ids) < 2:
continue
# Same shift as pretraining: input removes last token, target removes first token
x = torch.tensor(ids[:-1], device=device).unsqueeze(0)
y = torch.tensor(ids[1:], device=device).unsqueeze(0)
_, loss = model(x, y)
opt.zero_grad()
loss.backward()
opt.step()
Real SFT often uses loss masking, i.e., computing loss only on the answer part, not the question, to avoid wasting effort on learning to ask. Our data is small and short, so we simplify this away. Interested readers can look at
SFTTrainerin TRL.
After fine-tuning, modify the generation function to stop when it samples <|endoftext|>; otherwise it will keep writing after the answer. Then ask it:
Input: <|user|>小明喜欢什么?<|bot|>
Output: 小明喜欢编程,他每天早上都会写代码。<|endoftext|>
Two changes are visible: it no longer continues the question but directly outputs the answer, and it naturally stops at <|endoftext|>. A small model that can hold a conversation is now complete. Of course, it only answers what's in the corpus; asking out-of-scope questions makes it fabricate. For example, asking "What's the weather today?" yields something like "I'll say Chinese model", a fragment pieced together from training data.
Summary
Today we didn't learn any new concepts; we ran through every stage of the series by hand. The entire process requires very little compute: tokenizer training and corpus encoding are pure CPU work, and training the 7.5M model can be done on an ordinary development machine, no large-scale computing needed. Finally, let's summarize today's content:
- Corpus: We prepared a mixed Chinese-English toy corpus, 8 sentence patterns repeated 4000 times to make 314 KB, corresponding to the data engineering in Article 2.
- Tokenizer: We trained a BPE tokenizer with Hugging Face's tokenizers, configured with vocab 2048, but learned only 428 tokens, corresponding to Article 3.
- Model and Training: We hand-wrote a 4-layer, 7.5M-parameter mini GPT; the training loop used cross-entropy loss, AdamW, cosine annealing, and gradient clipping, corresponding to Articles 4 and 5.
- Memory: We observed the memory peak during training and demonstrated gradient accumulation for memory-constrained situations, corresponding to Article 6.
- Generation: Loss dropped from an initial 6.23 to near zero; the model memorized the entire toy corpus, transforming output from gibberish to full recitation.
- SFT: We fine-tuned with 12 instruction pairs for 300 steps; the model learned to answer in conversation format and stop automatically, corresponding to Article 9.
With this, our exploration of LLM training comes to an end. Looking back at the series, we started from a pile of internet text, went through tokenization, objective functions, backpropagation, optimizers, memory management, parallel strategies, pretraining, SFT, and preference alignment, eventually reaching a model that can converse. Today, we ran the entire thing by hand in a few hundred lines of code.
I hope this series helps you build a complete picture of LLM training: behind every seemingly magical capability is a chain of simple engineering. Training forges the model, inference liberates it. This series, together with the previous "LLM Inference" series, forms my complete learning record of the LLM tech stack; welcome to read them side by side.
Writing these two series is also my own learning journey. I'm a beginner myself; initially I understood nothing about LLM principles. I followed online tutorials and papers, had Claude Code teach me, asked questions when stuck, and verified with runnable examples. I moved forward little by little, but truly explaining each stage exposed many gaps. Reading papers is the most time-consuming, such as the few lines of derivation in DPO; the first read was completely baffling, but after hours of working through on paper, I finally understood why Z(x) cancels out—the satisfaction is unmatched by any secondhand interpretation. There are also many areas I still can't claim to truly understand; PPO's full implementation and TRL's source code I've read repeatedly, yet still only half-know.
Feynman learning says teaching is the best way to learn. Explaining something to others most exposes what you don't understand. This series is my practice of that method. I started in August with no knowledge, learning and writing along the way; it's been three months. There are inevitably places where the explanation is inadequate or even wrong. Readers are welcome to point them out for discussion; your corrections are also material for my continued learning.
References
- nanoGPT GitHub repository
- build-nanogpt: Replicating GPT-2 training
- nanochat GitHub repository
- minimind GitHub repository
- Happy-LLM GitHub repository
- train-llm-from-scratch GitHub repository
- LLMs-from-scratch: Companion code for "Build a Large Language Model from Scratch"
- Hugging Face tokenizers GitHub repository
- OpenAI tiktoken GitHub repository
- TRL GitHub repository
- PyTorch official beginner tutorial
Follow Us
If this article helped you, feel free to follow my WeChat public account: 日习一技,每天学一点新技术.
I spend an hour every day documenting my learning bits and pieces. Content includes, but is not limited to:
- Tips for using a product
- Practices and insights from open-source projects
- Simple explanations of technical points
The goal is to let you finish reading in 5 minutes and gain something without much effort, while easily picking up some dry goods. Whether you're a beginner or a veteran, feel free to give suggestions. If there's technology you want to learn, welcome to discuss!