Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.20016 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915501532250112 |
|---|---|
| author | Palattuparambil, Ajsal Shereef Karimpanal, Thommen George Rana, Santu |
| author_facet | Palattuparambil, Ajsal Shereef Karimpanal, Thommen George Rana, Santu |
| contents | Deep reinforcement learning (RL) policies, although optimal in terms of task rewards, may not align with the personal preferences of human users. To ensure this alignment, a naive solution would be to retrain the agent using a reward function that encodes the user's specific preferences. However, such a reward function is typically not readily available, and as such, retraining the agent from scratch can be prohibitively expensive. We propose a more practical approach - to adapt the already trained policy to user-specific needs with the help of human feedback. To this end, we infer the user's intent through trajectory-level feedback and combine it with the trained task policy via a theoretically grounded dynamic policy fusion approach. As our approach collects human feedback on the very same trajectories used to learn the task policy, it does not require any additional interactions with the environment, making it a zero-shot approach. We empirically demonstrate in a number of environments that our proposed dynamic policy fusion approach consistently achieves the intended task while simultaneously adhering to user-specific needs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_20016 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Dynamic Policy Fusion for User Alignment Without Re-Interaction Palattuparambil, Ajsal Shereef Karimpanal, Thommen George Rana, Santu Artificial Intelligence Machine Learning Deep reinforcement learning (RL) policies, although optimal in terms of task rewards, may not align with the personal preferences of human users. To ensure this alignment, a naive solution would be to retrain the agent using a reward function that encodes the user's specific preferences. However, such a reward function is typically not readily available, and as such, retraining the agent from scratch can be prohibitively expensive. We propose a more practical approach - to adapt the already trained policy to user-specific needs with the help of human feedback. To this end, we infer the user's intent through trajectory-level feedback and combine it with the trained task policy via a theoretically grounded dynamic policy fusion approach. As our approach collects human feedback on the very same trajectories used to learn the task policy, it does not require any additional interactions with the environment, making it a zero-shot approach. We empirically demonstrate in a number of environments that our proposed dynamic policy fusion approach consistently achieves the intended task while simultaneously adhering to user-specific needs. |
| title | Dynamic Policy Fusion for User Alignment Without Re-Interaction |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2409.20016 |