Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.27834 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911723485659136 |
|---|---|
| author | Hao, Guang-Yuan van der Laan, Lars Bibaut, Aurélien Kallus, Nathan |
| author_facet | Hao, Guang-Yuan van der Laan, Lars Bibaut, Aurélien Kallus, Nathan |
| contents | We study the transfer of rewards learned using inverse reinforcement learning from expert demonstrations in one environment to reinforcement learning in a new, different environment. This arises naturally when demonstrations are collected in a controlled environment. We formulate the problem as a joint system of Bellman equations across the source and target environments and develop minimax estimators for the target soft-$q$-function. Whereas a sequential solution approach first estimates the source reward and then plugs it into the target control problem, a coupled approach solves the source and target system of equations jointly. We show that, in contrast to the sequential approach, the coupled approach removes the first-order influence of source Bellman residual error. We characterize the local behavior of each approach, develop finite-sample soft-$q$-function error bounds, and prove regret guarantees for the resulting soft-control policy. An empirical investigation using a sepsis simulator validates the theoretical comparison. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_27834 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Reward Transfer from Inverse Reinforcement Learning: A Coupled Minimax Approach Hao, Guang-Yuan van der Laan, Lars Bibaut, Aurélien Kallus, Nathan Machine Learning We study the transfer of rewards learned using inverse reinforcement learning from expert demonstrations in one environment to reinforcement learning in a new, different environment. This arises naturally when demonstrations are collected in a controlled environment. We formulate the problem as a joint system of Bellman equations across the source and target environments and develop minimax estimators for the target soft-$q$-function. Whereas a sequential solution approach first estimates the source reward and then plugs it into the target control problem, a coupled approach solves the source and target system of equations jointly. We show that, in contrast to the sequential approach, the coupled approach removes the first-order influence of source Bellman residual error. We characterize the local behavior of each approach, develop finite-sample soft-$q$-function error bounds, and prove regret guarantees for the resulting soft-control policy. An empirical investigation using a sepsis simulator validates the theoretical comparison. |
| title | Reward Transfer from Inverse Reinforcement Learning: A Coupled Minimax Approach |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2605.27834 |