Perry Dong, a researcher at Stanford, and Chelsea Finn, a Stanford professor and co‑founder of robot company Physical Intelligence, have written about the obstacles that are preventing a major leap in robotics. They argue that while pretraining in robotics has scaled well, the field still lacks a shared, reliable approach to post‑training that would let models learn from their own experience.
What the language‑model community learned
Dong and Finn point out that the breakthrough for large language models (LLMs) came when the community converged on a shared post‑training recipe. They summarize that recipe in four steps:
- start with a strong pretrained model,
- define environments and rewards (for example, preference models),
- run reinforcement‑learning (RL) optimization against the reference model, and
- monitor for and address pathologies such as reward hacking.
They note that pretraining in robotics has scaled similarly, but the second half — enabling models to improve from their own experience — lacks a dependable, standardized solution.
What robotics needs
According to the authors, robotics needs an algorithm specifically designed to fine‑tune frontier robotics models: it must remain stable when applied to models with billions of parameters and learn from a small enough amount of experience to be practical on real hardware. Beyond the algorithm itself, the field must converge on a set of standard practices:
- a default definition of what counts as success,
- a default way to reset the scene between attempts so the robot can try again, and
- a default mechanism for a human to provide feedback and convert that feedback into learning.
These defaults are prerequisites for making post‑training broadly usable and scalable.
A concrete proposal: EXPO(-FT)
In their post, Dong and Finn describe a specific algorithm they have been developing, called EXPO(-FT), which they believe embodies some of these desired properties. The approach is still early and not widely adopted. EXPO(-FT) operates by repeatedly improving actions proposed by the frontier model using reinforcement learning with small edits from a lightweight policy, and then absorbing those improvements back into the frontier model itself.
Why this matters
Large, proprietary LLMs are already capable of giving instructions to robots and helping construct and execute complex plans. However, robots’ actual capabilities for movement and perception remain relatively primitive. Improving those capabilities will require models tailored to robot motion and vision, and standardization of post‑training practices like the ones Dong and Finn describe. They argue that language model post‑training became scalable because the field settled on concrete defaults; robotics is reaching a similar moment, and converging on industry defaults and a universal post‑training recipe is the crucial next step.
Conclusion
Dong and Finn conclude that while pretraining is established in robotics, the lack of a widely accepted, reliable post‑training algorithm is slowing progress. EXPO(-FT) is an early attempt to provide such a recipe, but broader agreement on defaults and practices will be necessary to scale improvements in robot movement and perception.
Further reading
Perry Dong’s blog post: “Towards Universal Post‑Training for Robotics”.
