This article is Sharon Zhou’s third post in a series about post-training; earlier parts covered foundations and details, and the final post in the series was scheduled for October 7. Here we review what post-training achieved for frontier models such as ChatGPT and the mechanisms that produced conversational, safer, tool-capable, and reasoning-oriented models.
Practical effects of post-training
Post-training turned pretrained language models like GPT-3 into systems that can reliably hold conversations, follow instructions, and interact with external tools. A clear illustration: prior to ChatGPT’s November 2022 launch, GPT-3 often produced nonsensical replies to simple prompts such as "Why do people like golden retrievers?" After the ChatGPT release, the same prompt typically returned a reasonable, human-like answer such as "Because they are affectionate, patient, and make excellent family pets," regardless of who typed it.
Post-training affects several capabilities:
-
Conversationality and helpfulness: models learn to ask clarifying questions when prompts are ambiguous, adjust tone and level of detail to the context (brief for quick facts, thorough for learning), and remain coherent across many conversational turns. They also become better at handling messy real-world inputs (for example, locating a clause inside a large PDF) without inventing answers.
-
Safety and alignment: post-training is the main instrument for teaching models to refuse harmful instructions, reduce biased or discriminatory outputs, avoid hallucinating when uncertain, and respect user privacy. The exact safety rules are set by the developers and can be taught in post-training to the extent that they can be encoded as reward signals.
-
Tool use and function calling: post-training trains models to recognize when external tools (search engines, APIs, calculators, databases, code interpreters) are useful, select the correct tool, construct proper API calls, interpret returned results, and present them conversationally. For example, a veterinary scheduling assistant trained for tool use would call a clinic’s scheduling API, check Dr. Patel’s availability, and reply: "Dr. Patel has an opening at 3:15pm today. I’ve tentatively held it for you. Should I confirm?" rather than inventing plausible but fictional appointment times.
-
Reasoning ("thinking"): post-training can encourage models to generate internal, multi-step chains of thought before producing a final answer. These reasoning models substantially outperform standard models on multistep logic tasks such as math problem solving, complex coding, scientific analysis, and planning.
Concrete result: on the 2024 AIME exam GPT-4o solved about 12% of problems on average, while OpenAI’s o1 reasoning model solved 74% on a single attempt; with 1,000 attempts and a learned reranking function it reached 93%, a score comparable to the top ~500 human test-takers.
The classic ChatGPT post-training pipeline (InstructGPT roots)
The pipeline that first brought global attention to ChatGPT drew on the InstructGPT paper. Although modern systems have evolved, this three-stage pipeline remains the conceptual foundation for many alignment methods:
- Supervised fine-tuning (SFT) on human demonstrations
- Training a reward model from human preference comparisons
- Reinforcement learning with human feedback (RLHF) to optimize the main model using the reward model
Stage 1: SFT on demonstrations
SFT teaches the model to follow instructions and behave like an assistant. OpenAI contracted roughly 40 human labelers, screened for skill at identifying harmful outputs. Labelers wrote ideal responses to prompts sourced from (1) real users who submitted prompts via the OpenAI API and (2) prompts the labelers themselves created (there were relatively few instruction-style API prompts before ChatGPT).
The SFT dataset contained about ~13,000 prompts with human-written responses. The base model at the time was GPT-3; they trained the SFT model for 16 epochs overall. Interestingly, the model tended to overfit after just one epoch in the SFT-only view, but SFT served as an intermediate phase and they chose checkpoints appropriate for the subsequent RLHF step. They also mixed in 10% pretraining data during this phase to help the next RL stage.
After SFT, the model could already converse and follow instructions—marked improvement over the pretrained GPT-3 checkpoint.
Stage 2: Preference data and reward modeling
The reward model must score millions of responses during RL training. OpenAI primarily trained the reward model on responses generated by the SFT model, but as the policy evolved during RL and produced new responses, the team continued to update the reward model with outputs from newer checkpoints to keep it robust.
To train the reward model, they collected pairwise preference comparisons. Instead of generating only two responses per prompt, they generated 4–9 responses and had human labelers rank them; rankings yield many pairwise comparisons (N-choose-2), so a dataset of 33K prompts with 4–9 responses each produced roughly 200K–1.2M pairwise comparisons for reward-model training. The reward model was kept small (6 billion parameters) for stability and efficiency, outputting a scalar reward.
Overfitting was a key challenge; training the reward model for only 1 epoch often helped. Because responses for a single prompt are highly correlated, they trained the pairs from the same prompt together in a batch and normalized them for stability and computational efficiency.
Stage 3: RLHF (reinforcement learning with human feedback)
With an SFT policy and a reward model in hand, RLHF continues to train the policy so it produces responses that the reward model scores highly. The algorithm used was PPO. The RLHF loop roughly proceeds as follows:
- Sample a prompt (OpenAI used a dataset of 31,000 prompts collected via the API).
- Generate a response (rollout) from the current policy.
- The reward model scores the full response. For every token in the response, the per-token reward is the reward-model score (applied at the final token) minus a per-token KL divergence penalty between the current policy and the original SFT model.
- The critic estimates expected rewards at each token position; advantage estimates (via GAE) compare actual per-token rewards to expected values.
- PPO updates the policy toward rollouts with higher advantage.
- Optionally mix in pretraining data and objective to avoid catastrophic forgetting, update the critic, and occasionally collect new preference data to retrain the reward model.
A crucial mechanism is the KL penalty between the RL policy and the SFT reference policy. Without it, the policy can drift into "reward hacking" behaviors—strange outputs that exploit quirks in the reward model to achieve high scores without being genuinely useful. The total reward combines the reward-model score minus the KL divergence multiplied by a coefficient that balances fidelity to the SFT policy versus reward maximization.
Training RLHF is computationally expensive because it runs multiple models simultaneously (policy, critic, reward, SFT reference) and requires fresh rollouts, reward scoring, advantage computation, and concurrent updates on every training step. Hyperparameters like learning rate, KL coefficient, rollout batch size, and clipping ratio are sensitive and can make many RL runs fail or produce degenerate behaviors.
Getting it right and empirical outcomes
The engineering complexity is high, but the improvements were clear: human evaluators preferred RLHF outputs over SFT outputs and SFT over the base pretrained model. In experiments, even a small 1.3 billion-parameter RLHF model was often preferred to a 175 billion-parameter SFT-only model, demonstrating that training method can matter as much as model size. Larger RLHF models still outperformed smaller RLHF models.
RLHF made models better at following explicit constraints, less likely to produce harmful outputs, and less prone to hallucinations, although it did not eliminate hallucinations entirely. One caveat: because labelers who created training data often evaluated models, the resulting models were partly optimized for that specific annotator population; held-out evaluators still preferred RLHF models but to a lesser degree.
Summary
Post-training—through SFT, reward modeling, and RLHF—was the key ingredient that transformed pretrained language models into the conversational, safer, tool-using, and reasoning-capable systems we now use. These methods bring substantial quality gains but also introduce significant computational cost, data and labeling requirements, and design choices about safety trade-offs that remain human decisions made in labs.
If you want deeper technical details on RLHF, reward-model design, or tool integration, I can expand on any of those components.
