Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.00388 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915335023624192 |
|---|---|
| author | Mu, Ni Hu, Hao Hu, Xiao Yang, Yiqin Xu, Bo Jia, Qing-Shan |
| author_facet | Mu, Ni Hu, Hao Hu, Xiao Yang, Yiqin Xu, Bo Jia, Qing-Shan |
| contents | Preference-based reinforcement learning (PbRL) bypasses explicit reward engineering by inferring reward functions from human preference comparisons, enabling better alignment with human intentions. However, humans often struggle to label a clear preference between similar segments, reducing label efficiency and limiting PbRL's real-world applicability. To address this, we propose an offline PbRL method: Contrastive LeArning for ResolvIng Ambiguous Feedback (CLARIFY), which learns a trajectory embedding space that incorporates preference information, ensuring clearly distinguished segments are spaced apart, thus facilitating the selection of more unambiguous queries. Extensive experiments demonstrate that CLARIFY outperforms baselines in both non-ideal teachers and real human feedback settings. Our approach not only selects more distinguished queries but also learns meaningful trajectory embeddings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_00388 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries Mu, Ni Hu, Hao Hu, Xiao Yang, Yiqin Xu, Bo Jia, Qing-Shan Machine Learning Preference-based reinforcement learning (PbRL) bypasses explicit reward engineering by inferring reward functions from human preference comparisons, enabling better alignment with human intentions. However, humans often struggle to label a clear preference between similar segments, reducing label efficiency and limiting PbRL's real-world applicability. To address this, we propose an offline PbRL method: Contrastive LeArning for ResolvIng Ambiguous Feedback (CLARIFY), which learns a trajectory embedding space that incorporates preference information, ensuring clearly distinguished segments are spaced apart, thus facilitating the selection of more unambiguous queries. Extensive experiments demonstrate that CLARIFY outperforms baselines in both non-ideal teachers and real human feedback settings. Our approach not only selects more distinguished queries but also learns meaningful trajectory embeddings. |
| title | CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2506.00388 |