MEReQ: Max-Ent Residual-Q Inverse RL for Sample-Efficient Alignment from Intervention
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915572468416512 |
|---|---|
| author | Chen, Yuxin Tang, Chen Wei, Jianglan Li, Chenran Tian, Ran Zhang, Xiang Zhan, Wei Stone, Peter Tomizuka, Masayoshi |
| author_facet | Chen, Yuxin Tang, Chen Wei, Jianglan Li, Chenran Tian, Ran Zhang, Xiang Zhan, Wei Stone, Peter Tomizuka, Masayoshi |
| contents | Aligning robot behavior with human preferences is crucial for deploying embodied AI agents in human-centered environments. A promising solution is interactive imitation learning from human intervention, where a human expert observes the policy's execution and provides interventions as feedback. However, existing methods often fail to utilize the prior policy efficiently to facilitate learning, thus hindering sample efficiency. In this work, we introduce MEReQ (Maximum-Entropy Residual-Q Inverse Reinforcement Learning), designed for sample-efficient alignment from human intervention. Instead of inferring the complete human behavior characteristics, MEReQ infers a residual reward function that captures the discrepancy between the human expert's and the prior policy's underlying reward functions. It then employs Residual Q-Learning (RQL) to align the policy with human preferences using this residual reward function. Extensive evaluations on simulated and real-world tasks demonstrate that MEReQ achieves sample-efficient policy alignment from human intervention. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_16258 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MEReQ: Max-Ent Residual-Q Inverse RL for Sample-Efficient Alignment from Intervention Chen, Yuxin Tang, Chen Wei, Jianglan Li, Chenran Tian, Ran Zhang, Xiang Zhan, Wei Stone, Peter Tomizuka, Masayoshi Robotics Artificial Intelligence Machine Learning I.2.6; I.2.9 Aligning robot behavior with human preferences is crucial for deploying embodied AI agents in human-centered environments. A promising solution is interactive imitation learning from human intervention, where a human expert observes the policy's execution and provides interventions as feedback. However, existing methods often fail to utilize the prior policy efficiently to facilitate learning, thus hindering sample efficiency. In this work, we introduce MEReQ (Maximum-Entropy Residual-Q Inverse Reinforcement Learning), designed for sample-efficient alignment from human intervention. Instead of inferring the complete human behavior characteristics, MEReQ infers a residual reward function that captures the discrepancy between the human expert's and the prior policy's underlying reward functions. It then employs Residual Q-Learning (RQL) to align the policy with human preferences using this residual reward function. Extensive evaluations on simulated and real-world tasks demonstrate that MEReQ achieves sample-efficient policy alignment from human intervention. |
| title | MEReQ: Max-Ent Residual-Q Inverse RL for Sample-Efficient Alignment from Intervention |
| topic | Robotics Artificial Intelligence Machine Learning I.2.6; I.2.9 |
| url | https://arxiv.org/abs/2406.16258 |