Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908524769968128 |
|---|---|
| author | Homayounirad, Amir Liscio, Enrico Wang, Tong Jonker, Catholijn M. Siebert, Luciano C. |
| author_facet | Homayounirad, Amir Liscio, Enrico Wang, Tong Jonker, Catholijn M. Siebert, Luciano C. |
| contents | Aggregating multiple annotations into a single ground truth label may hide valuable insights into annotator disagreement, particularly in tasks where subjectivity plays a crucial role. In this work, we explore methods for identifying subjectivity in recognizing the human values that motivate arguments. We evaluate two main approaches: inferring subjectivity through value prediction vs. directly identifying subjectivity. Our experiments show that direct subjectivity identification significantly improves the model performance of flagging subjective arguments. Furthermore, combining contrastive loss with binary cross-entropy loss does not improve performance but reduces the dependency on per-label subjectivity. Our proposed methods can help identify arguments that individuals may interpret differently, fostering a more nuanced annotation process. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_06704 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments Homayounirad, Amir Liscio, Enrico Wang, Tong Jonker, Catholijn M. Siebert, Luciano C. Computation and Language Aggregating multiple annotations into a single ground truth label may hide valuable insights into annotator disagreement, particularly in tasks where subjectivity plays a crucial role. In this work, we explore methods for identifying subjectivity in recognizing the human values that motivate arguments. We evaluate two main approaches: inferring subjectivity through value prediction vs. directly identifying subjectivity. Our experiments show that direct subjectivity identification significantly improves the model performance of flagging subjective arguments. Furthermore, combining contrastive loss with binary cross-entropy loss does not improve performance but reduces the dependency on per-label subjectivity. Our proposed methods can help identify arguments that individuals may interpret differently, fostering a more nuanced annotation process. |
| title | Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2509.06704 |