How LLMs Learn to Talk: Pre-training, From Next-Word Guessing to a Billion-Dollar Foundation

We use Doubao, DeepSeek, Qwen, Claude, and others every day... but they weren't originally "chatbots." They were text-completion machines that could only "continue writing." This machine is called a base model. Where does a base model come from? The answer is three words: pre-training.

What exactly does pre-training do?

Imagine locking someone in a library and making them read day and night. No one gives them tests or grades their homework; they just read the text and try to guess what the next word should be.

After a few years, they naturally learn grammar, common sense, logic, and can even write decent articles.

That is essentially what large model pre-training does.

Key point: No manual labeling is needed; the model learns patterns from massive amounts of text on its own.

Why is no labeling needed? Because the text itself is the answer. Let the model "guess the next word" or "fill in the masked word." If it guesses correctly, it has learned; if it guesses wrong, it adjusts its parameters. This training method, where the data acts as its own teacher, is called self-supervised learning.

---

What are pre-training, fine-tuning, and inference?

Many people mix up these three stages, but their division of labor is actually very clear:

Simply remember:

• Pre-training: Burying oneself in books, learning language and world knowledge.
• Fine-tuning: A teacher guides you through practice exams, learning to follow instructions.
• Inference: The user asks a question, and the model answers.

The ChatGPT we use daily became a conversational bot only after going through "pre-training + fine-tuning + reinforcement learning from human feedback." It wasn't born knowing how to chat.

---

Pre-training Methods

There are currently two mainstream pre-training methods:

School	Representative	Core Gameplay	What it's good at
Autoregressive	GPT series	Predict the next word	Text generation, dialogue, code completion
Masked Language Model	BERT series	Predict the masked word	Text understanding, classification, extraction

GPT is like a writing chain game; BERT is like a cloze test. The ChatGPT, Claude, and DeepSeek we see today all follow the GPT "chain game" path.

---

How is GPT trained?

Don't be intimidated by trillions of tokens; the principle is actually very simple.

The model receives each prefix and tries hard to guess what the next token is. Guessed wrong? No problem, calculate the error, backpropagate, adjust the parameters. After trillions of such guesses and corrections, the model gradually masters the patterns of language.

---

How is BERT trained?

BERT's method is more like the cloze tests teachers gave us in school:

It doesn't look at the following words; it can only infer the masked part based on the context. This method makes BERT very good at "understanding" text, rather than "generating" text.

---

Where does the training data come from?

In one sentence: Almost every piece of text that can be read.

Web pages, books, papers, code, Wikipedia, forum posts... as long as it's text, it can almost all be fed to the model. But raw data can't be consumed directly; it must be cleaned: deduplication, spam removal, filtering low-quality content, and privacy sanitization. Data cleaning often takes more effort than the training itself.

---

How much computing power does pre-training actually need?

What does the scale roughly look like?

• Data volume: Starting from trillions of tokens
• Model parameters: Billions to trillions
• GPU count: Thousands to tens of thousands of A100s/H100s
• Training time: Weeks to months

So pre-training is basically a game for big players; ordinary teams mostly fine-tune on top of open-source base models.

---

Scaling Law: Bigger is stronger, and it follows a rule

There is a famous discovery in the large model field called the Scaling Law:

Loss ≈ A / N^α + B / D^β + C

N = Number of parameters
D = Amount of data
α ≈ 0.076, β ≈ 0.095

Conclusion:
  Model size increases 10x → Data must also increase about 10x
  → Computation increases about 100x
  → Loss continues to decrease

In plain terms: As long as you dare to stack parameters, stack data, and stack computing power, the model's capability will steadily improve. This was the theoretical confidence that allowed OpenAI to scale GPT-3 to 175 billion parameters back then.

---

The Engineering Workflow of Pre-training

This isn't something you can run with a few lines of Python; it's a complete systems engineering project:

Distributed training is the main event. Data parallelism, tensor parallelism, and pipeline parallelism are used together to "spread" a trillion-parameter model across thousands of GPUs so it can run.

---

How expensive is pre-training?

• GPT-3 used roughly 355 GPU-years
• GPT-4 is rumored to be tens of thousands of GPU-years
• Electricity + hardware + manpower, total cost millions to billions of dollars

So pre-training is not just a competition of algorithms, but even more so of engineering, resources, and capital.

---

What does the model actually learn after pre-training?

The model after pre-training is usually called a Base Model. It is already very powerful, but it still "doesn't know how to chat."

It can complete text, translate, and write code, but if you ask it "Hello," it might reply with a Wikipedia-style definition instead of a natural greeting. To make it "know how to chat," it still needs subsequent SFT fine-tuning and RLHF human alignment.

---

Pre-training in the Entire Tech Stack

Pre-training is the foundation. The Transformer, Tokens, massive data, and Scaling Law together support it; on top of this foundation, fine-tuning and RLHF are needed to build the edifice of a conversational model.

---

The Full LLM Training Process

Comparing the three stages, you can understand why pre-training is the most expensive and has the longest cycle, while subsequent fine-tuning and alignment are relatively lightweight.

---

Can ordinary people participate in pre-training?

Seeing the thousands of GPUs, months of training, and millions of dollars mentioned earlier, many might think: Does this have anything to do with me?

The answer is: Yes, and the barrier to entry is lower than you think.

You indeed can't easily pre-train a GPT-4 from scratch, but that doesn't mean you can only be a spectator. There are at least three paths for ordinary people to enter the large model arena:

1. Take the fine-tuning route, stand on the shoulders of giants

There are already many open-source base models: Llama, Qwen, Mistral, DeepSeek-MoE... These models have already completed expensive pre-training, and you can directly take them for fine-tuning.

• Data: Prepare thousands to tens of thousands of domain-specific Q&A pairs
• Hardware: A few 3090s / A100s can run it
• Tools: Frameworks like LLaMA-Factory, Axolotl, and Unsloth have already packaged the process very simply

Spending a few days, you can create a custom model that "understands your business."

2. Do data engineering, an underestimated entry point

The quality of pre-training depends heavily on the data. Cleaning data, building high-quality corpora, and designing instruction datasets are precisely the areas where ordinary people can contribute the most.

• Crawlers + cleaning pipelines
• Deduplication, filtering, privacy sanitization
• Designing prompt templates, manual annotation, quality assessment

In the future, roles like "AI Trainer" and "Data Engineer" will only become more in demand.

3. Use models, don't train models

For most people, the most valuable way to participate is: Integrate large models into your own workflow.

Writing code, creating copy, processing spreadsheets, building knowledge bases, automating customer service... What truly sets people apart is not whether you can train a model, but whether you can use it better than others.

Pre-training is the battlefield for big companies, but application and fine-tuning are opportunities for everyone. You don't need to build a sports car from scratch; learning to drive, modify, and maintain it is enough to keep you ahead.

Summary

One-sentence summary:

Pre-training is a self-supervised learning process where a model does "cloze tests" or "predicts the next word" on massive amounts of text. It learns language rules and world knowledge, producing a base model; the base model then undergoes fine-tuning and human alignment to become the conversational ChatGPT, Claude, or DeepSeek we interact with.