The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913345340178432 |
|---|---|
| author | Fleisig, Eve Blodgett, Su Lin Klein, Dan Talat, Zeerak |
| author_facet | Fleisig, Eve Blodgett, Su Lin Klein, Dan Talat, Zeerak |
| contents | Longstanding data labeling practices in machine learning involve collecting and aggregating labels from multiple annotators. But what should we do when annotators disagree? Though annotator disagreement has long been seen as a problem to minimize, new perspectivist approaches challenge this assumption by treating disagreement as a valuable source of information. In this position paper, we examine practices and assumptions surrounding the causes of disagreement--some challenged by perspectivist approaches, and some that remain to be addressed--as well as practical and normative challenges for work operating under these assumptions. We conclude with recommendations for the data labeling pipeline and avenues for future research engaging with subjectivity and disagreement. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_05860 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels Fleisig, Eve Blodgett, Su Lin Klein, Dan Talat, Zeerak Machine Learning Computation and Language Computers and Society Longstanding data labeling practices in machine learning involve collecting and aggregating labels from multiple annotators. But what should we do when annotators disagree? Though annotator disagreement has long been seen as a problem to minimize, new perspectivist approaches challenge this assumption by treating disagreement as a valuable source of information. In this position paper, we examine practices and assumptions surrounding the causes of disagreement--some challenged by perspectivist approaches, and some that remain to be addressed--as well as practical and normative challenges for work operating under these assumptions. We conclude with recommendations for the data labeling pipeline and avenues for future research engaging with subjectivity and disagreement. |
| title | The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels |
| topic | Machine Learning Computation and Language Computers and Society |
| url | https://arxiv.org/abs/2405.05860 |