Convergence of a model-free entropy-regularized inverse reinforcement learning algorithm
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929739341496320 |
|---|---|
| author | Renard, Titouan Schlaginhaufen, Andreas Ni, Tingting Kamgarpour, Maryam |
| author_facet | Renard, Titouan Schlaginhaufen, Andreas Ni, Tingting Kamgarpour, Maryam |
| contents | Given a dataset of expert demonstrations, inverse reinforcement learning (IRL) aims to recover a reward for which the expert is optimal. This work proposes a model-free algorithm to solve entropy-regularized IRL problem. In particular, we employ a stochastic gradient descent update for the reward and a stochastic soft policy iteration update for the policy. Assuming access to a generative model, we prove that our algorithm is guaranteed to recover a reward for which the expert is $\varepsilon$-optimal using $\mathcal{O}(1/\varepsilon^{2})$ samples of the Markov decision process (MDP). Furthermore, with $\mathcal{O}(1/\varepsilon^{4})$ samples we prove that the optimal policy corresponding to the recovered reward is $\varepsilon$-close to the expert policy in total variation distance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_16829 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Convergence of a model-free entropy-regularized inverse reinforcement learning algorithm Renard, Titouan Schlaginhaufen, Andreas Ni, Tingting Kamgarpour, Maryam Machine Learning Artificial Intelligence Given a dataset of expert demonstrations, inverse reinforcement learning (IRL) aims to recover a reward for which the expert is optimal. This work proposes a model-free algorithm to solve entropy-regularized IRL problem. In particular, we employ a stochastic gradient descent update for the reward and a stochastic soft policy iteration update for the policy. Assuming access to a generative model, we prove that our algorithm is guaranteed to recover a reward for which the expert is $\varepsilon$-optimal using $\mathcal{O}(1/\varepsilon^{2})$ samples of the Markov decision process (MDP). Furthermore, with $\mathcal{O}(1/\varepsilon^{4})$ samples we prove that the optimal policy corresponding to the recovered reward is $\varepsilon$-close to the expert policy in total variation distance. |
| title | Convergence of a model-free entropy-regularized inverse reinforcement learning algorithm |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2403.16829 |