Don't Blame the Data, Blame the Model: Understanding Noise and Bias When Learning from Subjective Annotations
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914706140168192 |
|---|---|
| author | Anand, Abhishek Mokhberian, Negar Kumar, Prathyusha Naresh Saha, Anweasha He, Zihao Rao, Ashwin Morstatter, Fred Lerman, Kristina |
| author_facet | Anand, Abhishek Mokhberian, Negar Kumar, Prathyusha Naresh Saha, Anweasha He, Zihao Rao, Ashwin Morstatter, Fred Lerman, Kristina |
| contents | Researchers have raised awareness about the harms of aggregating labels especially in subjective tasks that naturally contain disagreements among human annotators. In this work we show that models that are only provided aggregated labels show low confidence on high-disagreement data instances. While previous studies consider such instances as mislabeled, we argue that the reason the high-disagreement text instances have been hard-to-learn is that the conventional aggregated models underperform in extracting useful signals from subjective tasks. Inspired by recent studies demonstrating the effectiveness of learning from raw annotations, we investigate classifying using Multiple Ground Truth (Multi-GT) approaches. Our experiments show an improvement of confidence for the high-disagreement instances. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_04085 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Don't Blame the Data, Blame the Model: Understanding Noise and Bias When Learning from Subjective Annotations Anand, Abhishek Mokhberian, Negar Kumar, Prathyusha Naresh Saha, Anweasha He, Zihao Rao, Ashwin Morstatter, Fred Lerman, Kristina Computation and Language Computers and Society Researchers have raised awareness about the harms of aggregating labels especially in subjective tasks that naturally contain disagreements among human annotators. In this work we show that models that are only provided aggregated labels show low confidence on high-disagreement data instances. While previous studies consider such instances as mislabeled, we argue that the reason the high-disagreement text instances have been hard-to-learn is that the conventional aggregated models underperform in extracting useful signals from subjective tasks. Inspired by recent studies demonstrating the effectiveness of learning from raw annotations, we investigate classifying using Multiple Ground Truth (Multi-GT) approaches. Our experiments show an improvement of confidence for the high-disagreement instances. |
| title | Don't Blame the Data, Blame the Model: Understanding Noise and Bias When Learning from Subjective Annotations |
| topic | Computation and Language Computers and Society |
| url | https://arxiv.org/abs/2403.04085 |