Jiaheng Hu, Jay Shim, and colleagues at the University of Texas Austin, University of California Los Angeles, Nanyang Technological University, and Sony report that combining low‑rank adaptation (LoRA) with on‑policy reinforcement learning (GRPO) when fine‑tuning large pretrained vision‑language‑action (VLA) models significantly reduces catastrophic forgetting in sequential robotics tasks. They evaluated the approach in simulation on the LIBERO benchmark and found it matched or exceeded prior continual‑learning baselines.
Experimental setup
The team fine‑tuned the OpenVLA‑OFT model on three task suites from the LIBERO benchmark, each suite containing five tasks (for example opening a drawer or moving an object to a target). Models were trained sequentially on each task without reusing data from earlier tasks. At each step the model received an image and an instruction and output a sequence of continuous control actions for a robot arm and gripper.
During fine‑tuning the authors applied LoRA to restrict how model weights changed, and they trained with GRPO, an on‑policy reinforcement learning method that rewards actions the model itself generated. The model received rewards for task completion during GRPO training.
Key insight
The authors argue that three elements together reduce forgetting: (1) large pretrained models whose many parameters make small updates less likely to overwrite prior knowledge, (2) LoRA, which constrains weight updates by representing changes as the product of two small matrices, and (3) on‑policy RL (GRPO), which limits updates by rewarding the model’s own actions. By contrast, supervised fine‑tuning and off‑policy RL can produce larger updates that interfere with previously learned behaviors.
Results
On the libero‑spatial tasks the method achieved an average success rate of 81.2%. Comparisons reported by the authors:
- Dark Experience Replay: 73.4% (reuses examples from earlier tasks),
- SLCA: 69.9% (higher learning rates in output layers, lower in early layers),
- Elastic Weight Consolidation: 66.1% (penalizes changes to weights important for prior tasks).
The authors’ recipe produced near‑zero forgetting: an average drop of only 0.3 percentage points in success rate on previously learned tasks. This is lower than Elastic Weight Consolidation (0.7) and Dark Experience Replay (4.7), and comparable to SLCA (−0.6, indicating a slight improvement on earlier tasks).
On five additional libero‑spatial tasks that were not used during training, the method reached 57.1% average success, outperforming Elastic Weight Consolidation (52.6%) and Dark Experience Replay (55.2%). The authors also report that removing any single component (large pretrained model, LoRA, or GRPO) caused performance to collapse and resulted in strong forgetting.
Caveats
For fairness the authors augmented earlier baseline methods with LoRA and GRPO on the LIBERO dataset for comparisons. However, those baselines were not necessarily designed to be combined with those techniques or to use the same data‑reuse protocol, so it remains unclear how they would perform under their originally intended settings. For example, Dark Experience Replay relies on reintroducing prior task examples during fine‑tuning; applying LoRA may change how a model learns new tasks and thus affect direct comparisons.
Why it matters
Training on all tasks at once can work well but requires prior knowledge of the full task set. When tasks arrive or change over time, sequential fine‑tuning is useful only if earlier skills are retained. The authors’ sequential approach (large pretrained models + LoRA + on‑policy RL) is simpler and, under the tested conditions, more effective than several prior methods for limiting catastrophic forgetting in robotics. The study does not evaluate whether the approach generalizes beyond robotics domains.
Conclusion
Combining LoRA with on‑policy reinforcement learning on large pretrained VLA models substantially reduces catastrophic forgetting in sequential robot task learning in the LIBERO simulation. The result is high task success, minimal forgetting, and improved generalization on some unseen tasks compared with several existing continual‑learning baselines.
