OpenAI · Artificial Intelligence

How LLMs Learn to Talk: Pre-training, From Next-Word Guessing to a Billion-Dollar Foundation

By 我是大卫 · Aug 10, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Pre-training is the capital-intensive foundation that every chatbot rests on, and understanding its cost structure and scaling behavior explains why the frontier is dominated by a handful of well-funded labs. For everyone else, the takeaway is practical: fine-tuning existing base models and doing data engineering are the accessible on-ramps.

Summary

Pre-training turns a blank-slate neural network into a base model by forcing it to guess the next token across trillions of words scraped from the web, books, and code. No human labels are needed: the text itself is the answer key, a method called self-supervised learning. GPT-style autoregressive models predict what comes next, while BERT-style masked models fill in hidden words, making them better at understanding than generation.

The compute bill is staggering. GPT-3 consumed roughly 355 GPU-years; GPT-4 is rumored to have required tens of thousands. Scaling laws show that loss drops predictably as parameters, data, and compute all increase together, which gave OpenAI the theoretical cover to push GPT-3 to 175 billion parameters. Distributed training across thousands of GPUs using data, tensor, and pipeline parallelism is the only way to make these runs feasible, and the whole process can take months and cost millions to billions of dollars.

What emerges is a base model that can complete text, translate, and write code but responds to "hello" with a Wikipedia definition. It takes supervised fine-tuning and RLHF to turn that raw capability into a conversational agent. For most developers, the practical path is not pre-training from scratch but fine-tuning open-weight base models like Llama or Qwen on a few GPUs, or building the data pipelines that make pre-training possible in the first place.

Takeaways

— Pre-training uses self-supervised learning: the model guesses the next word or a masked word, and the text itself provides the correct answer, so no manual labeling is needed.

— GPT-style autoregressive models predict the next token and excel at generation; BERT-style masked language models predict hidden words and excel at understanding.

— A base model fresh out of pre-training can complete text and write code but cannot hold a conversation; supervised fine-tuning and RLHF are required to turn it into a chatbot.

— Training data comes from nearly every readable source — web pages, books, code, Wikipedia — and must be deduplicated, filtered, and sanitized before use.

— Scaling laws describe a predictable relationship: loss decreases as a power law with more parameters and data, and a 10× increase in model size demands roughly 10× more data and 100× more compute.

— GPT-3 used about 355 GPU-years; GPT-4 is estimated at tens of thousands of GPU-years, putting total costs in the millions to billions of dollars.

— Distributed training combines data, tensor, and pipeline parallelism to spread trillion-parameter models across thousands of GPUs.

— Fine-tuning an open-source base model like Llama or Qwen on a few GPUs is the practical entry point for most teams, not pre-training from scratch.

— Data engineering — crawling, cleaning, deduplication, and instruction-dataset design — is an underrated and growing role in the LLM pipeline.

Conclusions

Scaling laws turned pre-training from an empirical gamble into an engineering discipline with predictable returns on investment, which is why labs keep pouring capital into larger runs despite the eye-watering costs.

The gap between a base model and a chatbot is a reminder that raw knowledge and conversational ability are separate capabilities; pre-training supplies the former, while alignment supplies the latter.

Data cleaning is often more labor-intensive than the training run itself, yet it receives far less attention than model architecture, even though it directly determines output quality.

The existence of capable open-weight base models has shifted the bottleneck from compute to data curation and evaluation, areas where individual developers can still make meaningful contributions.

Concepts & terms

Self-supervised learning

A training paradigm where the model generates its own labels from the input data — for example, by masking a word and trying to predict it — so no manual annotation is required.

Autoregressive language model

A model that generates text one token at a time, predicting each next token based on all previous tokens. GPT-style models are autoregressive.

Masked language model (MLM)

A model trained to predict randomly hidden words in a sentence using surrounding context. BERT is the canonical example, and it excels at understanding tasks rather than generation.

Scaling law

An empirical relationship showing that model loss decreases predictably as a power law with increases in parameters, data, and compute, giving a formula to estimate the return on larger training runs.

Base model

The raw model that emerges from pre-training. It can complete text and perform many tasks but has not been fine-tuned to follow instructions or hold conversations.

Distributed training parallelism

Techniques — data parallelism, tensor parallelism, and pipeline parallelism — that split a model and its training data across thousands of GPUs to make training trillion-parameter models feasible.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗