Enhancing Data Quality through Simple De-duplication: Navigating Responsible Computational Social Science Research
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929528074403840 |
|---|---|
| author | Mu, Yida Jin, Mali Song, Xingyi Aletras, Nikolaos |
| author_facet | Mu, Yida Jin, Mali Song, Xingyi Aletras, Nikolaos |
| contents | Research in natural language processing (NLP) for Computational Social Science (CSS) heavily relies on data from social media platforms. This data plays a crucial role in the development of models for analysing socio-linguistic phenomena within online communities. In this work, we conduct an in-depth examination of 20 datasets extensively used in NLP for CSS to comprehensively examine data quality. Our analysis reveals that social media datasets exhibit varying levels of data duplication. Consequently, this gives rise to challenges like label inconsistencies and data leakage, compromising the reliability of models. Our findings also suggest that data duplication has an impact on the current claims of state-of-the-art performance, potentially leading to an overestimation of model effectiveness in real-world scenarios. Finally, we propose new protocols and best practices for improving dataset development from social media data and its usage. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_03545 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhancing Data Quality through Simple De-duplication: Navigating Responsible Computational Social Science Research Mu, Yida Jin, Mali Song, Xingyi Aletras, Nikolaos Computation and Language Research in natural language processing (NLP) for Computational Social Science (CSS) heavily relies on data from social media platforms. This data plays a crucial role in the development of models for analysing socio-linguistic phenomena within online communities. In this work, we conduct an in-depth examination of 20 datasets extensively used in NLP for CSS to comprehensively examine data quality. Our analysis reveals that social media datasets exhibit varying levels of data duplication. Consequently, this gives rise to challenges like label inconsistencies and data leakage, compromising the reliability of models. Our findings also suggest that data duplication has an impact on the current claims of state-of-the-art performance, potentially leading to an overestimation of model effectiveness in real-world scenarios. Finally, we propose new protocols and best practices for improving dataset development from social media data and its usage. |
| title | Enhancing Data Quality through Simple De-duplication: Navigating Responsible Computational Social Science Research |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.03545 |