A Dataset for the Validation of Truth Inference Algorithms Suitable for Online Deployment
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913264443588608 |
|---|---|
| author | Wang, Fei Liu, Haoyu Bi, Haoyang Shen, Xiangzhuang Zhu, Renyu Wu, Runze Lin, Minmin Lv, Tangjie Fan, Changjie Liu, Qi Huang, Zhenya Chen, Enhong |
| author_facet | Wang, Fei Liu, Haoyu Bi, Haoyang Shen, Xiangzhuang Zhu, Renyu Wu, Runze Lin, Minmin Lv, Tangjie Fan, Changjie Liu, Qi Huang, Zhenya Chen, Enhong |
| contents | For the purpose of efficient and cost-effective large-scale data labeling, crowdsourcing is increasingly being utilized. To guarantee the quality of data labeling, multiple annotations need to be collected for each data sample, and truth inference algorithms have been developed to accurately infer the true labels. Despite previous studies having released public datasets to evaluate the efficacy of truth inference algorithms, these have typically focused on a single type of crowdsourcing task and neglected the temporal information associated with workers' annotation activities. These limitations significantly restrict the practical applicability of these algorithms, particularly in the context of long-term and online truth inference. In this paper, we introduce a substantial crowdsourcing annotation dataset collected from a real-world crowdsourcing platform. This dataset comprises approximately two thousand workers, one million tasks, and six million annotations. The data was gathered over a period of approximately six months from various types of tasks, and the timestamps of each annotation were preserved. We analyze the characteristics of the dataset from multiple perspectives and evaluate the effectiveness of several representative truth inference algorithms on this dataset. We anticipate that this dataset will stimulate future research on tracking workers' abilities over time in relation to different types of tasks, as well as enhancing online truth inference. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_08826 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Dataset for the Validation of Truth Inference Algorithms Suitable for Online Deployment Wang, Fei Liu, Haoyu Bi, Haoyang Shen, Xiangzhuang Zhu, Renyu Wu, Runze Lin, Minmin Lv, Tangjie Fan, Changjie Liu, Qi Huang, Zhenya Chen, Enhong Human-Computer Interaction Machine Learning For the purpose of efficient and cost-effective large-scale data labeling, crowdsourcing is increasingly being utilized. To guarantee the quality of data labeling, multiple annotations need to be collected for each data sample, and truth inference algorithms have been developed to accurately infer the true labels. Despite previous studies having released public datasets to evaluate the efficacy of truth inference algorithms, these have typically focused on a single type of crowdsourcing task and neglected the temporal information associated with workers' annotation activities. These limitations significantly restrict the practical applicability of these algorithms, particularly in the context of long-term and online truth inference. In this paper, we introduce a substantial crowdsourcing annotation dataset collected from a real-world crowdsourcing platform. This dataset comprises approximately two thousand workers, one million tasks, and six million annotations. The data was gathered over a period of approximately six months from various types of tasks, and the timestamps of each annotation were preserved. We analyze the characteristics of the dataset from multiple perspectives and evaluate the effectiveness of several representative truth inference algorithms on this dataset. We anticipate that this dataset will stimulate future research on tracking workers' abilities over time in relation to different types of tasks, as well as enhancing online truth inference. |
| title | A Dataset for the Validation of Truth Inference Algorithms Suitable for Online Deployment |
| topic | Human-Computer Interaction Machine Learning |
| url | https://arxiv.org/abs/2403.08826 |