Multi-Stage Manipulation with Demonstration-Augmented Reward, Policy, and World Model Learning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916986187939840 |
|---|---|
| author | Escoriza, Adrià López Hansen, Nicklas Tao, Stone Mu, Tongzhou Su, Hao |
| author_facet | Escoriza, Adrià López Hansen, Nicklas Tao, Stone Mu, Tongzhou Su, Hao |
| contents | Long-horizon tasks in robotic manipulation present significant challenges in reinforcement learning (RL) due to the difficulty of designing dense reward functions and effectively exploring the expansive state-action space. However, despite a lack of dense rewards, these tasks often have a multi-stage structure, which can be leveraged to decompose the overall objective into manageable subgoals. In this work, we propose DEMO3, a framework that exploits this structure for efficient learning from visual inputs. Specifically, our approach incorporates multi-stage dense reward learning, a bi-phasic training scheme, and world model learning into a carefully designed demonstration-augmented RL framework that strongly mitigates the challenge of exploration in long-horizon tasks. Our evaluations demonstrate that our method improves data-efficiency by an average of 40% and by 70% on particularly difficult tasks compared to state-of-the-art approaches. We validate this across 16 sparse-reward tasks spanning four domains, including challenging humanoid visual control tasks using as few as five demonstrations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01837 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Multi-Stage Manipulation with Demonstration-Augmented Reward, Policy, and World Model Learning Escoriza, Adrià López Hansen, Nicklas Tao, Stone Mu, Tongzhou Su, Hao Machine Learning Computer Vision and Pattern Recognition Robotics Long-horizon tasks in robotic manipulation present significant challenges in reinforcement learning (RL) due to the difficulty of designing dense reward functions and effectively exploring the expansive state-action space. However, despite a lack of dense rewards, these tasks often have a multi-stage structure, which can be leveraged to decompose the overall objective into manageable subgoals. In this work, we propose DEMO3, a framework that exploits this structure for efficient learning from visual inputs. Specifically, our approach incorporates multi-stage dense reward learning, a bi-phasic training scheme, and world model learning into a carefully designed demonstration-augmented RL framework that strongly mitigates the challenge of exploration in long-horizon tasks. Our evaluations demonstrate that our method improves data-efficiency by an average of 40% and by 70% on particularly difficult tasks compared to state-of-the-art approaches. We validate this across 16 sparse-reward tasks spanning four domains, including challenging humanoid visual control tasks using as few as five demonstrations. |
| title | Multi-Stage Manipulation with Demonstration-Augmented Reward, Policy, and World Model Learning |
| topic | Machine Learning Computer Vision and Pattern Recognition Robotics |
| url | https://arxiv.org/abs/2503.01837 |