Diverging Preferences: When do Annotators Disagree and do Models Know?
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917306566705152 |
|---|---|
| author | Zhang, Michael JQ Wang, Zhilin Hwang, Jena D. Dong, Yi Delalleau, Olivier Choi, Yejin Choi, Eunsol Ren, Xiang Pyatkin, Valentina |
| author_facet | Zhang, Michael JQ Wang, Zhilin Hwang, Jena D. Dong, Yi Delalleau, Olivier Choi, Yejin Choi, Eunsol Ren, Xiang Pyatkin, Valentina |
| contents | We examine diverging preferences in human-labeled preference datasets. We develop a taxonomy of disagreement sources spanning ten categories across four high-level classes and find that the majority of disagreements are due to factors such as task underspecification or response style. Our findings challenge a standard assumption in reward modeling methods that annotator disagreements can be attributed to simple noise. We then explore how these findings impact two areas of LLM development: reward modeling training and evaluation. In our experiments, we demonstrate how standard reward modeling (e.g., Bradley-Terry) and LLM-as-Judge evaluation methods fail to account for divergence between annotators. These findings highlight challenges in LLM evaluations, which are greatly influenced by divisive features like response style, and in developing pluralistically aligned LLMs. To address these issues, we develop methods for identifying diverging preferences to mitigate their influence in evaluations and during LLM training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_14632 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Diverging Preferences: When do Annotators Disagree and do Models Know? Zhang, Michael JQ Wang, Zhilin Hwang, Jena D. Dong, Yi Delalleau, Olivier Choi, Yejin Choi, Eunsol Ren, Xiang Pyatkin, Valentina Computation and Language We examine diverging preferences in human-labeled preference datasets. We develop a taxonomy of disagreement sources spanning ten categories across four high-level classes and find that the majority of disagreements are due to factors such as task underspecification or response style. Our findings challenge a standard assumption in reward modeling methods that annotator disagreements can be attributed to simple noise. We then explore how these findings impact two areas of LLM development: reward modeling training and evaluation. In our experiments, we demonstrate how standard reward modeling (e.g., Bradley-Terry) and LLM-as-Judge evaluation methods fail to account for divergence between annotators. These findings highlight challenges in LLM evaluations, which are greatly influenced by divisive features like response style, and in developing pluralistically aligned LLMs. To address these issues, we develop methods for identifying diverging preferences to mitigate their influence in evaluations and during LLM training. |
| title | Diverging Preferences: When do Annotators Disagree and do Models Know? |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.14632 |