Curriculum & Reward Learning for Robot Navigation and Locomotion
GACL and Reward Training Wheels — adaptive training for RL robots (IROS 2025)
Overview
Reinforcement learning for robots is bottlenecked by how we train, not just what we train: fixed task curricula and hand-designed rewards require expert effort and still leave performance on the table. This project develops two complementary frameworks — a first-author and a co-first-author paper at IROS 2025 — that adapt the training process itself to the robot’s learning progress.
Grounded Adaptive Curriculum Learning (GACL)
GACL (Wang et al., 2025) (with Zifan Xu, Peter Stone, and Xuesu Xiao) introduces a teacher-student paradigm in which an informed teacher generates training tasks by monitoring student performance in real time, while grounding the curriculum in limited reference samples from the target task distribution so training remains relevant to deployment.
The GACL teacher adapts task difficulty from live student performance while grounding tasks in the target distribution.
Published results: 6.8% higher success rate than state-of-the-art curriculum methods on wheeled navigation in constrained environments, and 6.1% higher on quadruped locomotion in challenging 3D confined spaces.
Reward Training Wheels (RTW)
RTW (Wang et al., 2025), co-first-authored with Tong Xu, automates auxiliary reward shaping: a teacher adaptively weights auxiliary reward components as the robot’s proficiency grows, so guidance fades out exactly like training wheels on a bicycle.
The RTW teacher observes the student's reward history and re-weights auxiliary rewards each iteration.
Published results: In simulation, RTW achieved a 2.35-point increase in navigation success and a 122.62% relative improvement in off-road mobility on vertically challenging terrain, reaching the respective performance thresholds about 35% and 3× faster. Sim-trained policies achieved 5/5 physical off-road trials versus 2/5 for expert-designed rewards, with up to 47.4% reduction in orientation angles (more stable poses).
Curriculum learning has emerged as a promising approach for training complex robotics tasks, yet current applications predominantly rely on manually designed curricula, which demand significant engineering effort and can suffer from subjective and suboptimal human design choices. While automated curriculum learning has shown success in simple domains like grid worlds and games where task distributions can be easily specified, robotics tasks present unique challenges: they require handling complex task spaces while maintaining relevance to target domain distributions that are only partially known through limited samples. To this end, we propose Grounded Adaptive Curriculum Learning, a framework specifically designed for robotics curriculum learning with three key innovations: (1) a task representation that consistently handles complex robot task design, (2) an active performance tracking mechanism that allows adaptive curriculum generation appropriate for the robot’s current capabilities, and (3) a grounding approach that maintains target domain relevance through alternating sampling between reference and synthetic tasks. We validate GACL on wheeled navigation in constrained environments and quadruped locomotion in challenging 3D confined spaces, achieving 6.8% and 6.1% higher success rates, respectively, than state-of-the-art methods in each domain.
@inproceedings{wang2025gacl,title={GACL: Grounded Adaptive Curriculum Learning with Active Task and Performance Monitoring},author={Wang, Linji and Xu, Zifan and Stone, Peter and Xiao, Xuesu},booktitle={2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},year={2025},pages={591--596},doi={10.1109/IROS60139.2025.11246910},}
Robotics Reinforcement Learning (RL) often relies on carefully engineered auxiliary rewards to supplement sparse primary learning objectives to compensate for the lack of large-scale, real-world, trial-and-error data. While these auxiliary rewards accelerate learning, they require significant engineering effort, may introduce human biases, and cannot adapt to the robot’s evolving capabilities during training. In this paper, we introduce Reward Training Wheels (RTW), a teacher-student framework that automates auxiliary reward adaptation for robotics RL. To be specific, the RTW teacher dynamically adjusts auxiliary reward weights based on the student’s evolving capabilities to determine which auxiliary reward aspects require more or less emphasis to improve the primary objective. We demonstrate RTW on two challenging robot tasks: navigation in highly constrained spaces and off-road vehicle mobility on vertically challenging terrain. In simulation, RTW outperforms expert-designed rewards by 2.35% in navigation success rate and improves off-road mobility performance by 122.62%, while achieving 35% and 3X faster training efficiency, respectively. Physical robot experiments further validate RTW’s effectiveness, achieving a perfect success rate (5/5 trials vs. 2/5 for expert-designed rewards) and improving vehicle stability with up to 47.4% reduction in orientation angles.
@inproceedings{wang2025reward,title={Reward Training Wheels: Adaptive Auxiliary Rewards for Robotics Reinforcement Learning},author={Wang, Linji and Xu, Tong and Lu, Yuanjie and Xiao, Xuesu},booktitle={2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},year={2025},pages={15262--15267},doi={10.1109/IROS60139.2025.11247039},}
