Your Policy Regularizer is Secretly an Adversary
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2022
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909182126456832 |
|---|---|
| author | Brekelmans, Rob Genewein, Tim Grau-Moya, Jordi Delétang, Grégoire Kunesch, Markus Legg, Shane Ortega, Pedro |
| author_facet | Brekelmans, Rob Genewein, Tim Grau-Moya, Jordi Delétang, Grégoire Kunesch, Markus Legg, Shane Ortega, Pedro |
| contents | Policy regularization methods such as maximum entropy regularization are widely used in reinforcement learning to improve the robustness of a learned policy. In this paper, we show how this robustness arises from hedging against worst-case perturbations of the reward function, which are chosen from a limited set by an imagined adversary. Using convex duality, we characterize this robust set of adversarial reward perturbations under KL and alpha-divergence regularization, which includes Shannon and Tsallis entropy regularization as special cases. Importantly, generalization guarantees can be given within this robust set. We provide detailed discussion of the worst-case reward perturbations, and present intuitive empirical examples to illustrate this robustness and its relationship with generalization. Finally, we discuss how our analysis complements and extends previous results on adversarial reward robustness and path consistency optimality conditions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2203_12592 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | Your Policy Regularizer is Secretly an Adversary Brekelmans, Rob Genewein, Tim Grau-Moya, Jordi Delétang, Grégoire Kunesch, Markus Legg, Shane Ortega, Pedro Machine Learning Policy regularization methods such as maximum entropy regularization are widely used in reinforcement learning to improve the robustness of a learned policy. In this paper, we show how this robustness arises from hedging against worst-case perturbations of the reward function, which are chosen from a limited set by an imagined adversary. Using convex duality, we characterize this robust set of adversarial reward perturbations under KL and alpha-divergence regularization, which includes Shannon and Tsallis entropy regularization as special cases. Importantly, generalization guarantees can be given within this robust set. We provide detailed discussion of the worst-case reward perturbations, and present intuitive empirical examples to illustrate this robustness and its relationship with generalization. Finally, we discuss how our analysis complements and extends previous results on adversarial reward robustness and path consistency optimality conditions. |
| title | Your Policy Regularizer is Secretly an Adversary |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2203.12592 |