Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911367207845888 |
|---|---|
| author | Ni, Jingwei Fan, Yu Zouhar, Vilém Rooein, Donya Hoyle, Alexander Sachan, Mrinmaya Leippold, Markus Hovy, Dirk Ash, Elliott |
| author_facet | Ni, Jingwei Fan, Yu Zouhar, Vilém Rooein, Donya Hoyle, Alexander Sachan, Mrinmaya Leippold, Markus Hovy, Dirk Ash, Elliott |
| contents | Variation in human annotation (i.e., disagreements) is common in NLP, often reflecting important information like task subjectivity and sample ambiguity. Modeling this variation is important for applications that are sensitive to such information. Although RLVR-style reasoning (Reinforcement Learning with Verifiable Rewards) has improved Large Language Model (LLM) performance on many tasks, it remains unclear whether such reasoning enables LLMs to capture informative variation in human annotation. In this work, we evaluate the influence of different reasoning settings on LLM disagreement modeling. We systematically evaluate each reasoning setting across model sizes, distribution expression methods, and steering methods, resulting in 60 experimental setups across 3 tasks. Surprisingly, our results show that RLVR-style reasoning degrades performance in disagreement modeling, while naive Chain-of-Thought (CoT) reasoning improves the performance of RLHF LLMs (RL from human feedback). These findings underscore the potential risk of replacing human annotators with reasoning LLMs, especially when disagreements are important. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_19467 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Can Reasoning Help Large Language Models Capture Human Annotator Disagreement? Ni, Jingwei Fan, Yu Zouhar, Vilém Rooein, Donya Hoyle, Alexander Sachan, Mrinmaya Leippold, Markus Hovy, Dirk Ash, Elliott Computation and Language Artificial Intelligence Variation in human annotation (i.e., disagreements) is common in NLP, often reflecting important information like task subjectivity and sample ambiguity. Modeling this variation is important for applications that are sensitive to such information. Although RLVR-style reasoning (Reinforcement Learning with Verifiable Rewards) has improved Large Language Model (LLM) performance on many tasks, it remains unclear whether such reasoning enables LLMs to capture informative variation in human annotation. In this work, we evaluate the influence of different reasoning settings on LLM disagreement modeling. We systematically evaluate each reasoning setting across model sizes, distribution expression methods, and steering methods, resulting in 60 experimental setups across 3 tasks. Surprisingly, our results show that RLVR-style reasoning degrades performance in disagreement modeling, while naive Chain-of-Thought (CoT) reasoning improves the performance of RLHF LLMs (RL from human feedback). These findings underscore the potential risk of replacing human annotators with reasoning LLMs, especially when disagreements are important. |
| title | Can Reasoning Help Large Language Models Capture Human Annotator Disagreement? |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2506.19467 |