Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909556579237888 |
|---|---|
| author | Lopez-Cardona, Angela Segura, Carlos Karatzoglou, Alexandros Abadal, Sergi Arapakis, Ioannis |
| author_facet | Lopez-Cardona, Angela Segura, Carlos Karatzoglou, Alexandros Abadal, Sergi Arapakis, Ioannis |
| contents | Advancements in Natural Language Processing (NLP), have led to the emergence of Large Language Models (LLMs) such as GPT, Llama, Claude, and Gemini, which excel across a range of tasks but require extensive fine-tuning to align their outputs with human expectations. A widely used method for achieving this alignment is Reinforcement Learning from Human Feedback (RLHF), which, despite its success, faces challenges in accurately modelling human preferences. In this paper, we introduce GazeReward, a novel framework that integrates implicit feedback -- and specifically eye-tracking (ET) data -- into the Reward Model (RM). In addition, we explore how ET-based features can provide insights into user preferences. Through ablation studies we test our framework with different integration methods, LLMs, and ET generator models, demonstrating that our approach significantly improves the accuracy of the RM on established human preference datasets. This work advances the ongoing discussion on optimizing AI alignment with human values, exploring the potential of cognitive data for shaping future NLP research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_01532 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Models Lopez-Cardona, Angela Segura, Carlos Karatzoglou, Alexandros Abadal, Sergi Arapakis, Ioannis Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Human-Computer Interaction Advancements in Natural Language Processing (NLP), have led to the emergence of Large Language Models (LLMs) such as GPT, Llama, Claude, and Gemini, which excel across a range of tasks but require extensive fine-tuning to align their outputs with human expectations. A widely used method for achieving this alignment is Reinforcement Learning from Human Feedback (RLHF), which, despite its success, faces challenges in accurately modelling human preferences. In this paper, we introduce GazeReward, a novel framework that integrates implicit feedback -- and specifically eye-tracking (ET) data -- into the Reward Model (RM). In addition, we explore how ET-based features can provide insights into user preferences. Through ablation studies we test our framework with different integration methods, LLMs, and ET generator models, demonstrating that our approach significantly improves the accuracy of the RM on established human preference datasets. This work advances the ongoing discussion on optimizing AI alignment with human values, exploring the potential of cognitive data for shaping future NLP research. |
| title | Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Models |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Human-Computer Interaction |
| url | https://arxiv.org/abs/2410.01532 |