Aligning Human Intent from Imperfect Demonstrations with Confidence-based Inverse soft-Q Learning
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929412299030528 |
|---|---|
| author | Bu, Xizhou Li, Wenjuan Liu, Zhengxiong Ma, Zhiqiang Huang, Panfeng |
| author_facet | Bu, Xizhou Li, Wenjuan Liu, Zhengxiong Ma, Zhiqiang Huang, Panfeng |
| contents | Imitation learning attracts much attention for its ability to allow robots to quickly learn human manipulation skills through demonstrations. However, in the real world, human demonstrations often exhibit random behavior that is not intended by humans. Collecting high-quality human datasets is both challenging and expensive. Consequently, robots need to have the ability to learn behavioral policies that align with human intent from imperfect demonstrations. Previous work uses confidence scores to extract useful information from imperfect demonstrations, which relies on access to ground truth rewards or active human supervision. In this paper, we propose a transition-based method to obtain fine-grained confidence scores for data without the above efforts, which can increase the success rate of the baseline algorithm by 40.3$\%$ on average. We develop a generalized confidence-based imitation learning framework for guiding policy learning, called Confidence-based Inverse soft-Q Learning (CIQL), as shown in Fig.1. Based on this, we analyze two ways of processing noise and find that penalization is more aligned with human intent than filtering. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_11194 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Aligning Human Intent from Imperfect Demonstrations with Confidence-based Inverse soft-Q Learning Bu, Xizhou Li, Wenjuan Liu, Zhengxiong Ma, Zhiqiang Huang, Panfeng Robotics Imitation learning attracts much attention for its ability to allow robots to quickly learn human manipulation skills through demonstrations. However, in the real world, human demonstrations often exhibit random behavior that is not intended by humans. Collecting high-quality human datasets is both challenging and expensive. Consequently, robots need to have the ability to learn behavioral policies that align with human intent from imperfect demonstrations. Previous work uses confidence scores to extract useful information from imperfect demonstrations, which relies on access to ground truth rewards or active human supervision. In this paper, we propose a transition-based method to obtain fine-grained confidence scores for data without the above efforts, which can increase the success rate of the baseline algorithm by 40.3$\%$ on average. We develop a generalized confidence-based imitation learning framework for guiding policy learning, called Confidence-based Inverse soft-Q Learning (CIQL), as shown in Fig.1. Based on this, we analyze two ways of processing noise and find that penalization is more aligned with human intent than filtering. |
| title | Aligning Human Intent from Imperfect Demonstrations with Confidence-based Inverse soft-Q Learning |
| topic | Robotics |
| url | https://arxiv.org/abs/2312.11194 |