Manning has published Reinforcement Learning from Human Feedback: Aligning and Post‑training LLMs, a book that compiles several years of practical lessons about post‑training large language models and the application of reinforcement learning (RL) to alignment. The book aims to give intuitive explanations for why post‑training methods work, what trade‑offs they involve, and which misconceptions commonly impede correct application.
Origins and motivation
The work began as a website where the author documented post‑training methods that lacked clear online explanations. Topics such as rejection sampling, outcome reward models, and character training were often not covered in a foundational, intuitive way; the site became a popular resource for exactly those gaps. The book expands those materials into a unified narrative.
What the book covers
Structurally, the book mixes history, algorithmic derivations, and practical guidance. About 25% of the content focuses directly on reinforcement learning. That RL portion is intended to build the reader’s intuition about how different RL algorithms change model outputs and when a new algorithm may be promising versus unlikely to succeed.
Algorithms and topics treated in detail include:
- Deriving the Policy Gradient and Policy Gradient theory
- Vanilla Policy Gradient and REINFORCE
- REINFORCE Leave One Out (RLOO)
- Proximal Policy Optimization (PPO) and understanding the PPO objective
- Value functions and their relation to PPO
- Group Relative Policy Optimization (GRPO)
- Group Sequence Policy Optimization (GSPO)
- Clipped Importance Sampling Policy Optimization (CISPO)
- Algorithm comparisons
The book presents hands‑on intuitions—for example, explaining PPO clipping by showing how a single sample’s position in the surrogate objective’s regions determines whether gradients flow or are clipped. These intuitions feed directly into system design choices to handle gradients and numerical issues.
Systems design and engineering trade‑offs
Modern RL is often a systems engineering problem balancing how off‑policy the data are, training‑to‑inference mismatch, and throughput. The author describes the common asynchronous RL architecture in which separate GPUs serve learners (which apply gradient updates) and actors (which generate rollouts). Agentic tasks add infrastructure on top of these fundamentals.
Practical material includes loss aggregation trade‑offs (which inspired early DAPO and GRPO variants), truncated importance sampling used in early PPO experiments, and concrete code patterns and examples. The book provides simple formulae and examples such as the general RL loss form:
pg_loss = -advantages * ratio
Historical perspective
A dedicated chapter (Chapter 2) traces the field’s evolution: early work on preference‑based RL up to around 2018; the period of adapting RL to language models from 2019 to 2022; and the 2023‑onward era influenced by examples such as ChatGPT. The book stresses the contributions of early researchers who laid the foundations for later scale‑up.
Distillation and clearing up misconceptions
Chapters 10–12 aim to make opaque data practices more transparent. Chapter 12 is a textbook‑style treatment of distillation, explaining how LLM outputs are used to train downstream models and how industry practice evolved from the original 2015 knowledge‑distillation literature to multi‑teacher on‑policy distillation (MOPD) approaches used in models like Xiaomi MiMo‑V2‑Flash and DeepSeek V4. The goal is to demystify the broad — and sometimes politically charged — use of the term “distillation.”
Common failure modes in post‑training
The second half of the book surveys many practical headaches: over‑optimization, regularization, evaluation, and character training. It explains, for example, why RL‑trained models may generalize where supervised fine‑tuning (SFT) forgets (at the mathematical level), how frontier labs shape model personalities, and why such shaping can overshoot. The book presents tools and then walks through the practical challenges practitioners will meet when applying them.
The author also argues that because frontier research can land in production models within 3–9 months today (rather than years as before), companies must pay more attention to relevant research; the book trains readers to recognize which research matters.
Extras, availability and discount
The book is available freely online and is accompanied by a full 12‑hour course (slides and video on YouTube), a simple code base with suggested exercises, and model completion comparisons. Manning is offering 50% off until August 19 with the code PBLambert. Physical copies are shipping from Manning and Amazon US now, and from Amazon UK in October.
The author notes that the book’s title may be slightly out of date due to lessons learned during publication, but emphasizes the content is up to date; an on‑policy distillation section was added at the last moment. The book is aimed at readers with at least a bachelor’s degree in computer science; mastering it should give a substantial advantage in understanding post‑training and RLHF.
