Researchers at Carnegie Mellon University — Yuxiao Qu, Amrith Setlur, Virginia Smith, Ruslan Salakhutdinov and Aviral Kumar — proposed Privileged On-Policy Exploration (POPE), a training scheme that combines the on-policy reinforcement learning algorithm GRPO with custom datasets that include the opening steps of solutions as privileged hints. The method appends short solution prefixes to problems during training so the model can discover how to continue from a promising intermediate state.
Why this approach?
Exploration is a major bottleneck in reinforcement learning: when a model cannot stumble upon all required steps, it cannot learn to solve hard problems. Supervised fine-tuning on full solutions can lead to memorizing specific solutions rather than learning general problem-solving strategies. POPE uses the initial steps of a solution as guidance so the model can more easily find a complete solution, and, by training on both hinted and unhinted versions, it can learn to reproduce the early steps itself.
Method details
- The authors fine-tuned a pretrained Qwen3-4B-Instruct-2507 model using GRPO and a customized dataset.
- From three math datasets, they selected examples that the pretrained model failed to solve correctly in 128 attempts; each attempt allowed generating up to 32,000 tokens.
- For every selected example they extracted the start of the solution (a prefix) and provided progressively longer prefixes — up to one quarter of the full solution length — until the model could correctly complete the solution.
- They appended that prefix to the corresponding problem along with an instruction to continue solving from that point.
- During GRPO training, each problem was shown to the model many times, in equal proportion with and without its prefix. When the model solved a problem, GRPO increased the probability of the tokens it produced; if it failed, GRPO decreased those probabilities.
Results
The authors compared Qwen3-4B-Instruct-2507 after POPE fine-tuning to standard GRPO and to supervised fine-tuning. POPE consistently outperformed both baselines and did so by a substantial margin over supervised fine-tuning. Evaluation metrics were pass@1 (one try) and pass@16 (16 tries).
- On the AIME 2025 competition math dataset, POPE achieved 53.1% pass@1 and 82.6% pass@16, compared with typical GRPO at 49.6% pass@1 and 81.4% pass@16.
- On HMMT 2025, POPE reached 37.8% pass@1 and 67.5% pass@16, versus typical GRPO at 31.0% pass@1 and 63.8% pass@16.
Limitations
POPE depends on having tasks with known solutions from which to extract prefixes. In domains where obtaining such solutions is expensive or impractical, POPE inherits that cost.
Why it matters
This work addresses exploration inefficiency in reinforcement learning by guiding models to promising intermediate states rather than relying on blind search. By breaking learning into (i) finding a good starting state and (ii) solving from that state, POPE first teaches models to complete a plausible prefix and thereby makes it easier for them to later learn how to reach those prefixes on their own.
