By Sharon Zhou
This is the first article in a series about post‑training.
Why post‑training was needed
Early large language models (LLMs) accumulated vast amounts of raw knowledge but were often practically unusable for end users. Post‑training is the set of supervised and reinforcement techniques that teaches models specific behaviors: how to hold natural conversations, call external tools (for example, a calculator or search API), respond more safely, and tackle complex tasks by thinking step‑by‑step (so‑called reasoning).
Behavior here is broader than polite responses — post‑training enables a model to recognize and respond to particular token patterns (such as chat templates), to invoke tools, and to validate outputs programmatically.
From GPT‑3 to ChatGPT: the post‑training revolution
GPT‑3 appeared in June 2020. It was a 175 billion parameter completion engine trained to predict next tokens from web pages, articles, and documents. Because its pretraining data did not primarily contain conversation transcripts, it often produced statistically‑likely continuations rather than human‑like answers. That could produce listicle‑style or otherwise odd outputs.
At the time, practitioners relied on prompt engineering and few‑shot examples to coax GPT‑3 into answering questions in the desired form. While this could work, it was brittle: tiny changes in the prompt — for example, adding a space after “A:” — change tokenization and the model’s statistical future, often resulting in drastically different outputs.
Post‑training made models robust to the intended token patterns and conversational formats used in products. After post‑training, models were far more reliable for chat applications and broader product use, which helped convince many that AI had undergone a practical paradigm shift.
Pretraining versus post‑training
Pretraining: raw intelligence
- Pretraining optimizes the model to predict the next token over a very large, curated dataset. It is largely unsupervised and supplies factual knowledge (history, programming syntax, mathematics).
- Researchers run thousands of experiments to find the right data mix, architectures, and hyperparameters. The final full‑scale run (sometimes called a “god run”) can take over a month on thousands of GPUs.
- Performance is commonly measured with metrics like perplexity: lower perplexity is better; a perplexity near 1 indicates very low uncertainty per token.
Post‑training: shaping behavior
- Post‑training works on targeted datasets: human demonstrations of ideal responses, human judgments ranking model outputs, and programmatic scorers that define what “good” looks like.
- Objectives in post‑training (for example, “be more helpful” or “avoid harmful outputs”) are inherently more subjective than next‑token prediction. Loss curves are noisier and data quality matters more.
- The compute scale of standard post‑training is usually modest — tens to hundreds of GPUs for days — enabling rapid iteration. However, training models for sophisticated reasoning can make post‑training account for a large fraction of the model’s total compute, sometimes approaching the cost and engineering complexity of pretraining.
Why post‑training matters: usability, safety, capability
Usability
A pretrained model is like handing someone a giant PDF of Wikipedia: the information is there but not easy to query or use. Post‑training teaches the model how to present and retrieve that information in natural conversation, maintain longer multi‑turn interactions, and follow instructions so ordinary users can interact without prompt‑engineering expertise.
Safety
Pretraining data often contains toxic, biased, or dangerous material. A model will follow patterns in its data unless guided otherwise. Post‑training and additional guardrails teach models to refuse harmful requests, avoid generating inappropriate content, and express uncertainty when appropriate (for example, “I don’t know” or “consult a professional”).
Alignment — the broader research goal of aligning models with human values and preferences — is primarily pursued through post‑training. Companies also layer lightweight safety models to screen inputs and outputs as extra protection.
Capability
Post‑training can improve performance on hard tasks. A pretrained model might have the factual knowledge needed to solve a complex math word problem but default to an incorrect answer because of pattern matching. By encouraging the model to output more tokens before giving a final answer (i.e., to “think longer”), post‑training can produce more logically sound solutions. Reasoning improvements now map directly to business value for companies deploying these models.
Superhuman performance and limits
Post‑training has helped models achieve superhuman results in specific domains. Top reasoning models can solve competitive programming problems at levels exceeding most human competitors, and in mathematics they have reached scores comparable to top Math Olympiad performers. In some scientific areas, models have proposed novel hypotheses and solutions that experts found useful.
Two factors enable this. First, pretraining aggregates knowledge from millions of sources that no single human has read. Second, post‑training — especially verifier‑based reinforcement learning — lets models explore vastly more solution approaches than a human would in one sitting. The real limitation becomes how well we can define and provide reward signals that encode what “good” means.
Closing thoughts
Pretraining provides raw capability; post‑training determines how that capability manifests in practice: tone, clarity, safety, tool use, and deliberate reasoning. As research focuses more on post‑training methods — particularly reasoning and verification techniques — models become more useful in products, but this also raises computation and engineering costs.
Further reading: Richard Ngo, Lawrence Chan, and Sören Mindermann, “The Alignment Problem from a Deep Learning Perspective”; Iason Gabriel, “Artificial Intelligence, Values, and Alignment.”
