Boosting Robustness in Preference-Based Reinforcement Learning with Dynamic Sparsity
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916824641175552 |
|---|---|
| author | Muslimani, Calarina Grooten, Bram Mamillapalli, Deepak Ranganatha Sastry Pechenizkiy, Mykola Mocanu, Decebal Constantin Taylor, Matthew E. |
| author_facet | Muslimani, Calarina Grooten, Bram Mamillapalli, Deepak Ranganatha Sastry Pechenizkiy, Mykola Mocanu, Decebal Constantin Taylor, Matthew E. |
| contents | To integrate into human-centered environments, autonomous agents must learn from and adapt to humans in their native settings. Preference-based reinforcement learning (PbRL) can enable this by learning reward functions from human preferences. However, humans live in a world full of diverse information, most of which is irrelevant to completing any particular task. It then becomes essential that agents learn to focus on the subset of task-relevant state features. To that end, this work proposes R2N (Robust-to-Noise), the first PbRL algorithm that leverages principles of dynamic sparse training to learn robust reward models that can focus on task-relevant features. In experiments with a simulated teacher, we demonstrate that R2N can adapt the sparse connectivity of its neural networks to focus on task-relevant features, enabling R2N to significantly outperform several sparse training and PbRL algorithms across simulated robotic environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_06495 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Boosting Robustness in Preference-Based Reinforcement Learning with Dynamic Sparsity Muslimani, Calarina Grooten, Bram Mamillapalli, Deepak Ranganatha Sastry Pechenizkiy, Mykola Mocanu, Decebal Constantin Taylor, Matthew E. Machine Learning To integrate into human-centered environments, autonomous agents must learn from and adapt to humans in their native settings. Preference-based reinforcement learning (PbRL) can enable this by learning reward functions from human preferences. However, humans live in a world full of diverse information, most of which is irrelevant to completing any particular task. It then becomes essential that agents learn to focus on the subset of task-relevant state features. To that end, this work proposes R2N (Robust-to-Noise), the first PbRL algorithm that leverages principles of dynamic sparse training to learn robust reward models that can focus on task-relevant features. In experiments with a simulated teacher, we demonstrate that R2N can adapt the sparse connectivity of its neural networks to focus on task-relevant features, enabling R2N to significantly outperform several sparse training and PbRL algorithms across simulated robotic environments. |
| title | Boosting Robustness in Preference-Based Reinforcement Learning with Dynamic Sparsity |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2406.06495 |