Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.08012 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917774732820480 |
|---|---|
| author | Ovinnikov, Ivan Bykovets, Eugene Buhmann, Joachim M. |
| author_facet | Ovinnikov, Ivan Bykovets, Eugene Buhmann, Joachim M. |
| contents | Inverse reinforcement learning methods aim to retrieve the reward function of a Markov decision process based on a dataset of expert demonstrations. The commonplace scarcity and heterogeneous sources of such demonstrations can lead to the absorption of spurious correlations in the data by the learned reward function. Consequently, this adaptation often exhibits behavioural overfitting to the expert data set when a policy is trained on the obtained reward function under distribution shift of the environment dynamics. In this work, we explore a novel regularization approach for inverse reinforcement learning methods based on the causal invariance principle with the goal of improved reward function generalization. By applying this regularization to both exact and approximate formulations of the learning task, we demonstrate superior policy performance when trained using the recovered reward functions in a transfer setting |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_08012 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Learning Causally Invariant Reward Functions from Diverse Demonstrations Ovinnikov, Ivan Bykovets, Eugene Buhmann, Joachim M. Machine Learning Artificial Intelligence Inverse reinforcement learning methods aim to retrieve the reward function of a Markov decision process based on a dataset of expert demonstrations. The commonplace scarcity and heterogeneous sources of such demonstrations can lead to the absorption of spurious correlations in the data by the learned reward function. Consequently, this adaptation often exhibits behavioural overfitting to the expert data set when a policy is trained on the obtained reward function under distribution shift of the environment dynamics. In this work, we explore a novel regularization approach for inverse reinforcement learning methods based on the causal invariance principle with the goal of improved reward function generalization. By applying this regularization to both exact and approximate formulations of the learning task, we demonstrate superior policy performance when trained using the recovered reward functions in a transfer setting |
| title | Learning Causally Invariant Reward Functions from Diverse Demonstrations |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2409.08012 |