Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917479723302912 |
|---|---|
| author | Belay, Tadesse Destaw Ahmad, Ibrahim Said Abdulmumin, Idris Ayele, Abinew Ali Gelbukh, Alexander Ricárdez-Vázquez, Eusebio Kolesnikova, Olga Muhammad, Shamsuddeen Hassan Yimam, Seid Muhie |
| author_facet | Belay, Tadesse Destaw Ahmad, Ibrahim Said Abdulmumin, Idris Ayele, Abinew Ali Gelbukh, Alexander Ricárdez-Vázquez, Eusebio Kolesnikova, Olga Muhammad, Shamsuddeen Hassan Yimam, Seid Muhie |
| contents | Disagreement in annotation is a common phenomenon in the development of NLP datasets and serves as a valuable source of insight. While majority voting remains the dominant strategy for aggregating labels, recent work has explored modeling individual annotators to preserve their perspectives. However, modeling each annotator is resource-intensive and remains underexplored across various NLP tasks. We propose an agreement-based clustering technique to model the disagreement between the annotators. We conduct comprehensive experiments in 40 datasets in 18 typologically diverse languages, covering three subjective NLP tasks: sentiment analysis, emotion classification, and hate speech detection. We evaluate four aggregation approaches: majority vote, ensemble, multi-label, and multitask. The results demonstrate that agreement-based clustering can leverage the full spectrum of annotator perspectives and significantly enhance classification performance in subjective NLP tasks compared to majority voting and individual annotator modeling. Regarding the aggregation approach, the multi-label and multitask approaches are better for modeling clustered annotators than an ensemble and model majority vote. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_09955 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks Belay, Tadesse Destaw Ahmad, Ibrahim Said Abdulmumin, Idris Ayele, Abinew Ali Gelbukh, Alexander Ricárdez-Vázquez, Eusebio Kolesnikova, Olga Muhammad, Shamsuddeen Hassan Yimam, Seid Muhie Computation and Language Disagreement in annotation is a common phenomenon in the development of NLP datasets and serves as a valuable source of insight. While majority voting remains the dominant strategy for aggregating labels, recent work has explored modeling individual annotators to preserve their perspectives. However, modeling each annotator is resource-intensive and remains underexplored across various NLP tasks. We propose an agreement-based clustering technique to model the disagreement between the annotators. We conduct comprehensive experiments in 40 datasets in 18 typologically diverse languages, covering three subjective NLP tasks: sentiment analysis, emotion classification, and hate speech detection. We evaluate four aggregation approaches: majority vote, ensemble, multi-label, and multitask. The results demonstrate that agreement-based clustering can leverage the full spectrum of annotator perspectives and significantly enhance classification performance in subjective NLP tasks compared to majority voting and individual annotator modeling. Regarding the aggregation approach, the multi-label and multitask approaches are better for modeling clustered annotators than an ensemble and model majority vote. |
| title | Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.09955 |