OASST-ETC Dataset: Alignment Signals from Eye-tracking Analysis of LLM Responses
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918043850899456 |
|---|---|
| author | Lopez-Cardona, Angela Idesis, Sebastian Barreda-Ángeles, Miguel Abadal, Sergi Arapakis, Ioannis |
| author_facet | Lopez-Cardona, Angela Idesis, Sebastian Barreda-Ángeles, Miguel Abadal, Sergi Arapakis, Ioannis |
| contents | While Large Language Models (LLMs) have significantly advanced natural language processing, aligning them with human preferences remains an open challenge. Although current alignment methods rely primarily on explicit feedback, eye-tracking (ET) data offers insights into real-time cognitive processing during reading. In this paper, we present OASST-ETC, a novel eye-tracking corpus capturing reading patterns from 24 participants, while evaluating LLM-generated responses from the OASST1 dataset. Our analysis reveals distinct reading patterns between preferred and non-preferred responses, which we compare with synthetic eye-tracking data. Furthermore, we examine the correlation between human reading measures and attention patterns from various transformer-based models, discovering stronger correlations in preferred responses. This work introduces a unique resource for studying human cognitive processing in LLM evaluation and suggests promising directions for incorporating eye-tracking data into alignment methods. The dataset and analysis code are publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_10927 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | OASST-ETC Dataset: Alignment Signals from Eye-tracking Analysis of LLM Responses Lopez-Cardona, Angela Idesis, Sebastian Barreda-Ángeles, Miguel Abadal, Sergi Arapakis, Ioannis Computation and Language Artificial Intelligence While Large Language Models (LLMs) have significantly advanced natural language processing, aligning them with human preferences remains an open challenge. Although current alignment methods rely primarily on explicit feedback, eye-tracking (ET) data offers insights into real-time cognitive processing during reading. In this paper, we present OASST-ETC, a novel eye-tracking corpus capturing reading patterns from 24 participants, while evaluating LLM-generated responses from the OASST1 dataset. Our analysis reveals distinct reading patterns between preferred and non-preferred responses, which we compare with synthetic eye-tracking data. Furthermore, we examine the correlation between human reading measures and attention patterns from various transformer-based models, discovering stronger correlations in preferred responses. This work introduces a unique resource for studying human cognitive processing in LLM evaluation and suggests promising directions for incorporating eye-tracking data into alignment methods. The dataset and analysis code are publicly available. |
| title | OASST-ETC Dataset: Alignment Signals from Eye-tracking Analysis of LLM Responses |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2503.10927 |