Exploring the Influence of Label Aggregation on Minority Voices: Implications for Dataset Bias and Model Training
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929616849993728 |
|---|---|
| author | Pandya, Mugdha Moosavi, Nafise Sadat Maynard, Diana |
| author_facet | Pandya, Mugdha Moosavi, Nafise Sadat Maynard, Diana |
| contents | Resolving disagreement in manual annotation typically consists of removing unreliable annotators and using a label aggregation strategy such as majority vote or expert opinion to resolve disagreement. These may have the side-effect of silencing or under-representing minority but equally valid opinions. In this paper, we study the impact of standard label aggregation strategies on minority opinion representation in sexism detection. We investigate the quality and value of minority annotations, and then examine their effect on the class distributions in gold labels, as well as how this affects the behaviour of models trained on the resulting datasets. Finally, we discuss the potential biases introduced by each method and how they can be amplified by the models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_04025 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Exploring the Influence of Label Aggregation on Minority Voices: Implications for Dataset Bias and Model Training Pandya, Mugdha Moosavi, Nafise Sadat Maynard, Diana Computation and Language Resolving disagreement in manual annotation typically consists of removing unreliable annotators and using a label aggregation strategy such as majority vote or expert opinion to resolve disagreement. These may have the side-effect of silencing or under-representing minority but equally valid opinions. In this paper, we study the impact of standard label aggregation strategies on minority opinion representation in sexism detection. We investigate the quality and value of minority annotations, and then examine their effect on the class distributions in gold labels, as well as how this affects the behaviour of models trained on the resulting datasets. Finally, we discuss the potential biases introduced by each method and how they can be amplified by the models. |
| title | Exploring the Influence of Label Aggregation on Minority Voices: Implications for Dataset Bias and Model Training |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.04025 |