Expert Proximity as Surrogate Rewards for Single Demonstration Imitation Learning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914860328026112 |
|---|---|
| author | Chiang, Chia-Cheng Lan, Li-Cheng Sun, Wei-Fang Feng, Chien Hsieh, Cho-Jui Lee, Chun-Yi |
| author_facet | Chiang, Chia-Cheng Lan, Li-Cheng Sun, Wei-Fang Feng, Chien Hsieh, Cho-Jui Lee, Chun-Yi |
| contents | In this paper, we focus on single-demonstration imitation learning (IL), a practical approach for real-world applications where acquiring multiple expert demonstrations is costly or infeasible and the ground truth reward function is not available. In contrast to typical IL settings with multiple demonstrations, single-demonstration IL involves an agent having access to only one expert trajectory. We highlight the issue of sparse reward signals in this setting and propose to mitigate this issue through our proposed Transition Discriminator-based IL (TDIL) method. TDIL is an IRL method designed to address reward sparsity by introducing a denser surrogate reward function that considers environmental dynamics. This surrogate reward function encourages the agent to navigate towards states that are proximal to expert states. In practice, TDIL trains a transition discriminator to differentiate between valid and non-valid transitions in a given environment to compute the surrogate rewards. The experiments demonstrate that TDIL outperforms existing IL approaches and achieves expert-level performance in the single-demonstration IL setting across five widely adopted MuJoCo benchmarks as well as the "Adroit Door" robotic environment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_01057 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Expert Proximity as Surrogate Rewards for Single Demonstration Imitation Learning Chiang, Chia-Cheng Lan, Li-Cheng Sun, Wei-Fang Feng, Chien Hsieh, Cho-Jui Lee, Chun-Yi Machine Learning In this paper, we focus on single-demonstration imitation learning (IL), a practical approach for real-world applications where acquiring multiple expert demonstrations is costly or infeasible and the ground truth reward function is not available. In contrast to typical IL settings with multiple demonstrations, single-demonstration IL involves an agent having access to only one expert trajectory. We highlight the issue of sparse reward signals in this setting and propose to mitigate this issue through our proposed Transition Discriminator-based IL (TDIL) method. TDIL is an IRL method designed to address reward sparsity by introducing a denser surrogate reward function that considers environmental dynamics. This surrogate reward function encourages the agent to navigate towards states that are proximal to expert states. In practice, TDIL trains a transition discriminator to differentiate between valid and non-valid transitions in a given environment to compute the surrogate rewards. The experiments demonstrate that TDIL outperforms existing IL approaches and achieves expert-level performance in the single-demonstration IL setting across five widely adopted MuJoCo benchmarks as well as the "Adroit Door" robotic environment. |
| title | Expert Proximity as Surrogate Rewards for Single Demonstration Imitation Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2402.01057 |