Modern post-training for large language models (LLMs) relies primarily on two methods: reinforcement learning (RL) and supervised fine-tuning (SFT). Both approaches have deep roots in machine learning, but their application to shaping LLM behavior forms the backbone of contemporary post-training. Nearly all post-training work is some blend of these two techniques.
Reinforcement learning (RL): learning from feedback
In RL the loop is: the model receives a prompt, generates a response, the response is scored (a reward), and the model’s weights are updated so higher-reward responses become more likely. A central question is: where does the reward come from?
-
Verifiers: automated functions provide rewards, for example a checker that runs tests on generated code or validates a math solution. Ideal verifiers are fast, cheap, and reliable within their domain, but they only work where “correctness” can be programmatically defined. Many desirable attributes — helpfulness, nuance, pleasant tone — are not verifiable this way.
-
Human feedback and RLHF: humans provide rich, subtle preference signals, but having humans judge every training example is infeasible. The InstructGPT work introduced reinforcement learning from human feedback (RLHF) by training a separate reward model to mimic human judgments. Pairwise comparisons ("Which of these two responses is better?") are commonly used because they yield higher interannotator agreement than absolute scores. InstructGPT used roughly 33,000 prompts; showing labelers 4–9 outputs per prompt turns each ranking into many pairwise comparisons, which translates into roughly 200,000–1,200,000 comparisons for that scale.
-
LLM-as-judge (RLAIF): an LLM can act as the judge to produce reward signals, a technique sometimes called reinforcement learning from AI feedback (RLAIF). This scales better than human annotation and can evaluate subjective qualities, but it carries the judge model’s biases and can be gamed. A robust approach is to break judgment into multiple LLM calls across dimensions (factual accuracy, clarity, tone, completeness), score each, and combine them with weights. Anthropic’s Constitutional AI (CAI) uses written principles (a "constitution") to have an LLM critique and rank its own outputs; those AI-generated comparisons then train a reward model.
-
RL algorithms: once you have rewards, algorithms convert them into training signals. Important variants include:
- REINFORCE: conceptually simple — reward a generated response proportionally — but noisy and high-variance.
- PPO (proximal policy optimization): used in OpenAI’s original ChatGPT work and for a time the standard for RLHF. PPO clips updates to prevent large parameter shifts, improving stability. It’s an online algorithm where the model continually generates responses, gets them graded by a reward model, and updates. PPO also uses a critic model to predict expected rewards and compute advantages, which reduces variance but adds complexity and infrastructure overhead.
- DPO (direct preference optimization): avoids an RL loop by directly training on pairwise preference data with supervised learning. Theoretically DPO and PPO-based RLHF can converge to the same optimum under ideal conditions, but in practice differences arise (DPO is offline and cannot explore beyond its fixed dataset). Its simplicity makes DPO attractive for smaller teams.
- GRPO (group relative policy optimization): introduced by DeepSeek, GRPO removes the critic by generating a group of responses to the same prompt and using the group average as the baseline. It’s simpler than PPO but maintains online exploration capability; it was used in training DeepSeek’s reasoning models.
RL is generally less stable than supervised learning: loss curves are noisier, hyperparameters are sensitive, and training can diverge without careful management. Practitioners commonly constrain RL updates with a KL-divergence penalty to keep the fine-tuned model close to the base or SFT model, acting as a regularizer.
Supervised fine-tuning (SFT): teaching by demonstration
SFT is straightforward: collect {prompt, ideal response} pairs and continue training the model on those with the next-token prediction objective, computing loss only on response tokens. Its advantages are simplicity and stability: there’s no reward model, critic, or policy-gradient variance. Its limitation is dependence on data quality and scale.
-
Human demonstrations: hiring skilled annotators to write high-quality responses is the gold standard. It provides precise control — InstructGPT contracted 40 labelers to write demonstrations and rank outputs — but it is expensive and can be inconsistent at scale.
-
Synthetic data: LLMs can generate millions of demonstrations cheaply. The Stanford Alpaca project fine-tuned Llama on 52,000 demonstrations generated by text-davinci-003 and achieved qualitatively similar behavior to the source model on a narrow evaluation (~250 examples). Such approaches scale well but may face legal or terms-of-service constraints depending on the source model’s policies.
-
Curated data with synthetic transformations: existing human-generated content (support logs, documentation, expert Q&A) can be transformed into prompt–response pairs using LLM pipelines. This grounds the data in real use cases but requires heavy cleaning and curation.
-
Rejection sampling: generate many outputs for a prompt, score them with a quality metric (test suite, reward model, or stronger judge), and keep only the best responses for SFT. This technique—used, for example, in Meta’s Llama 2 post-training pipeline—leverages the model’s best outputs to raise its average performance. Its limit is the model’s sampling ceiling: if the model never produces a correct solution, filtering can’t create one.
SFT also has shortcomings: it teaches only what to do and not what not to do, so problematic outputs can persist on underrepresented prompts. Mixed-style training data can induce mode averaging, producing responses with awkward blended tones.
Why frontier models use both techniques
SFT is stable and reliable, giving models good baseline behavior and serving as a warm start for RL. RL can push models to discover novel reasoning and rare but important behaviors, and it scales well for tasks where verifiers can be programmatic (for example, math problems). For instance, DeepSeek found that RL applied to a strong pretrained model (DeepSeek R1-Zero) yielded powerful reasoning but left usability issues like language mixing.
In practice, a standard pipeline often looks like:
- Pretraining to produce a foundation model.
- SFT to teach basic behaviors: multi-turn dialogue, instruction following, helpful tone.
- RL to refine the SFT checkpoint using reward signals from human preferences, verifiers, or AI judges, improving consistency, safety, and complex-task performance.
Teams frequently iterate between SFT and RL—SFT, RL, then more SFT on new data, then further RL—depending on the task. More compute is often devoted to RL than SFT, but SFT examples remain critical for stabilizing later RL stages.
Conclusion
Reinforcement learning and supervised fine-tuning are complementary pillars of modern LLM post-training. SFT delivers stable, dependable behavior and easier implementation; RL uncovers new capabilities, handles adversarial or edge-case prompts, and can scale with programmatic verifiers. Most state-of-the-art systems combine both: SFT to bring the model into a usable space, and RL to hone it toward human preferences and harder, frontier-level performance. The specific balance depends on available data, compute, and the desired use cases.
