A Guide to Misinformation Detection Data and Evaluation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915422662557696 |
|---|---|
| author | Thibault, Camille Tian, Jacob-Junqi Peloquin-Skulski, Gabrielle Curtis, Taylor Lynn Zhou, James Laflamme, Florence Guan, Yuxiang Rabbany, Reihaneh Godbout, Jean-François Pelrine, Kellin |
| author_facet | Thibault, Camille Tian, Jacob-Junqi Peloquin-Skulski, Gabrielle Curtis, Taylor Lynn Zhou, James Laflamme, Florence Guan, Yuxiang Rabbany, Reihaneh Godbout, Jean-François Pelrine, Kellin |
| contents | Misinformation is a complex societal issue, and mitigating solutions are difficult to create due to data deficiencies. To address this, we have curated the largest collection of (mis)information datasets in the literature, totaling 75. From these, we evaluated the quality of 36 datasets that consist of statements or claims, as well as the 9 datasets that consist of data in purely paragraph form. We assess these datasets to identify those with solid foundations for empirical work and those with flaws that could result in misleading and non-generalizable results, such as spurious correlations, or examples that are ambiguous or otherwise impossible to assess for veracity. We find the latter issue is particularly severe and affects most datasets in the literature. We further provide state-of-the-art baselines on all these datasets, but show that regardless of label quality, categorical labels may no longer give an accurate evaluation of detection model performance. Finally, we propose and highlight Evaluation Quality Assurance (EQA) as a tool to guide the field toward systemic solutions rather than inadvertently propagating issues in evaluation. Overall, this guide aims to provide a roadmap for higher quality data and better grounded evaluations, ultimately improving research in misinformation detection. All datasets and other artifacts are available at https://misinfo-datasets.complexdatalab.com/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_05060 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Guide to Misinformation Detection Data and Evaluation Thibault, Camille Tian, Jacob-Junqi Peloquin-Skulski, Gabrielle Curtis, Taylor Lynn Zhou, James Laflamme, Florence Guan, Yuxiang Rabbany, Reihaneh Godbout, Jean-François Pelrine, Kellin Social and Information Networks Computation and Language Computers and Society Misinformation is a complex societal issue, and mitigating solutions are difficult to create due to data deficiencies. To address this, we have curated the largest collection of (mis)information datasets in the literature, totaling 75. From these, we evaluated the quality of 36 datasets that consist of statements or claims, as well as the 9 datasets that consist of data in purely paragraph form. We assess these datasets to identify those with solid foundations for empirical work and those with flaws that could result in misleading and non-generalizable results, such as spurious correlations, or examples that are ambiguous or otherwise impossible to assess for veracity. We find the latter issue is particularly severe and affects most datasets in the literature. We further provide state-of-the-art baselines on all these datasets, but show that regardless of label quality, categorical labels may no longer give an accurate evaluation of detection model performance. Finally, we propose and highlight Evaluation Quality Assurance (EQA) as a tool to guide the field toward systemic solutions rather than inadvertently propagating issues in evaluation. Overall, this guide aims to provide a roadmap for higher quality data and better grounded evaluations, ultimately improving research in misinformation detection. All datasets and other artifacts are available at https://misinfo-datasets.complexdatalab.com/. |
| title | A Guide to Misinformation Detection Data and Evaluation |
| topic | Social and Information Networks Computation and Language Computers and Society |
| url | https://arxiv.org/abs/2411.05060 |