D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917641972613120 |
|---|---|
| author | Davani, Aida Mostafazadeh Díaz, Mark Baker, Dylan Prabhakaran, Vinodkumar |
| author_facet | Davani, Aida Mostafazadeh Díaz, Mark Baker, Dylan Prabhakaran, Vinodkumar |
| contents | While human annotations play a crucial role in language technologies, annotator subjectivity has long been overlooked in data collection. Recent studies that have critically examined this issue are often situated in the Western context, and solely document differences across age, gender, or racial groups. As a result, NLP research on subjectivity have overlooked the fact that individuals within demographic groups may hold diverse values, which can influence their perceptions beyond their group norms. To effectively incorporate these considerations into NLP pipelines, we need datasets with extensive parallel annotations from various social and cultural groups. In this paper we introduce the \dataset dataset: a large-scale cross-cultural dataset of parallel annotations for offensive language in over 4.5K sentences annotated by a pool of over 4k annotators, balanced across gender and age, from across 21 countries, representing eight geo-cultural regions. The dataset contains annotators' moral values captured along six moral foundations: care, equality, proportionality, authority, loyalty, and purity. Our analyses reveal substantial regional variations in annotators' perceptions that are shaped by individual moral values, offering crucial insights for building pluralistic, culturally sensitive NLP models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_10857 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation Davani, Aida Mostafazadeh Díaz, Mark Baker, Dylan Prabhakaran, Vinodkumar Computation and Language While human annotations play a crucial role in language technologies, annotator subjectivity has long been overlooked in data collection. Recent studies that have critically examined this issue are often situated in the Western context, and solely document differences across age, gender, or racial groups. As a result, NLP research on subjectivity have overlooked the fact that individuals within demographic groups may hold diverse values, which can influence their perceptions beyond their group norms. To effectively incorporate these considerations into NLP pipelines, we need datasets with extensive parallel annotations from various social and cultural groups. In this paper we introduce the \dataset dataset: a large-scale cross-cultural dataset of parallel annotations for offensive language in over 4.5K sentences annotated by a pool of over 4k annotators, balanced across gender and age, from across 21 countries, representing eight geo-cultural regions. The dataset contains annotators' moral values captured along six moral foundations: care, equality, proportionality, authority, loyalty, and purity. Our analyses reveal substantial regional variations in annotators' perceptions that are shaped by individual moral values, offering crucial insights for building pluralistic, culturally sensitive NLP models. |
| title | D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2404.10857 |