Interactive Groupwise Comparison for Reinforcement Learning from Human Feedback
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915639701012480 |
|---|---|
| author | Kompatscher, Jan Shi, Danqing Varni, Giovanna Weinkauf, Tino Oulasvirta, Antti |
| author_facet | Kompatscher, Jan Shi, Danqing Varni, Giovanna Weinkauf, Tino Oulasvirta, Antti |
| contents | Reinforcement learning from human feedback (RLHF) has emerged as a key enabling technology for aligning AI behaviour with human preferences. The traditional way to collect data in RLHF is via pairwise comparisons: human raters are asked to indicate which one of two samples they prefer. We present an interactive visualisation that better exploits the human visual ability to compare and explore whole groups of samples. The interface is comprised of two linked views: 1) an exploration view showing a contextual overview of all sampled behaviours organised in a hierarchical clustering structure; and 2) a comparison view displaying two selected groups of behaviours for user queries. Users can efficiently explore large sets of behaviours by iterating between these two views. Additionally, we devised an active learning approach suggesting groups for comparison. As shown by our evaluation in six simulated robotics tasks, our approach increases the final rewards by 69.34%. It leads to lower error rates and better policies. We open-source the code that can be easily integrated into the RLHF training loop, supporting research on human-AI alignment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_04340 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Interactive Groupwise Comparison for Reinforcement Learning from Human Feedback Kompatscher, Jan Shi, Danqing Varni, Giovanna Weinkauf, Tino Oulasvirta, Antti Machine Learning Human-Computer Interaction I.2.6; H.5.2; I.2.9 Reinforcement learning from human feedback (RLHF) has emerged as a key enabling technology for aligning AI behaviour with human preferences. The traditional way to collect data in RLHF is via pairwise comparisons: human raters are asked to indicate which one of two samples they prefer. We present an interactive visualisation that better exploits the human visual ability to compare and explore whole groups of samples. The interface is comprised of two linked views: 1) an exploration view showing a contextual overview of all sampled behaviours organised in a hierarchical clustering structure; and 2) a comparison view displaying two selected groups of behaviours for user queries. Users can efficiently explore large sets of behaviours by iterating between these two views. Additionally, we devised an active learning approach suggesting groups for comparison. As shown by our evaluation in six simulated robotics tasks, our approach increases the final rewards by 69.34%. It leads to lower error rates and better policies. We open-source the code that can be easily integrated into the RLHF training loop, supporting research on human-AI alignment. |
| title | Interactive Groupwise Comparison for Reinforcement Learning from Human Feedback |
| topic | Machine Learning Human-Computer Interaction I.2.6; H.5.2; I.2.9 |
| url | https://arxiv.org/abs/2507.04340 |