Wikibench: Community-Driven Data Curation for AI Evaluation on Wikipedia
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916133809946624 |
|---|---|
| author | Kuo, Tzu-Sheng Halfaker, Aaron Cheng, Zirui Kim, Jiwoo Wu, Meng-Hsin Wu, Tongshuang Holstein, Kenneth Zhu, Haiyi |
| author_facet | Kuo, Tzu-Sheng Halfaker, Aaron Cheng, Zirui Kim, Jiwoo Wu, Meng-Hsin Wu, Tongshuang Holstein, Kenneth Zhu, Haiyi |
| contents | AI tools are increasingly deployed in community contexts. However, datasets used to evaluate AI are typically created by developers and annotators outside a given community, which can yield misleading conclusions about AI performance. How might we empower communities to drive the intentional design and curation of evaluation datasets for AI that impacts them? We investigate this question on Wikipedia, an online community with multiple AI-based content moderation tools deployed. We introduce Wikibench, a system that enables communities to collaboratively curate AI evaluation datasets, while navigating ambiguities and differences in perspective through discussion. A field study on Wikipedia shows that datasets curated using Wikibench can effectively capture community consensus, disagreement, and uncertainty. Furthermore, study participants used Wikibench to shape the overall data curation process, including refining label definitions, determining data inclusion criteria, and authoring data statements. Based on our findings, we propose future directions for systems that support community-driven data curation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_14147 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Wikibench: Community-Driven Data Curation for AI Evaluation on Wikipedia Kuo, Tzu-Sheng Halfaker, Aaron Cheng, Zirui Kim, Jiwoo Wu, Meng-Hsin Wu, Tongshuang Holstein, Kenneth Zhu, Haiyi Human-Computer Interaction Artificial Intelligence AI tools are increasingly deployed in community contexts. However, datasets used to evaluate AI are typically created by developers and annotators outside a given community, which can yield misleading conclusions about AI performance. How might we empower communities to drive the intentional design and curation of evaluation datasets for AI that impacts them? We investigate this question on Wikipedia, an online community with multiple AI-based content moderation tools deployed. We introduce Wikibench, a system that enables communities to collaboratively curate AI evaluation datasets, while navigating ambiguities and differences in perspective through discussion. A field study on Wikipedia shows that datasets curated using Wikibench can effectively capture community consensus, disagreement, and uncertainty. Furthermore, study participants used Wikibench to shape the overall data curation process, including refining label definitions, determining data inclusion criteria, and authoring data statements. Based on our findings, we propose future directions for systems that support community-driven data curation. |
| title | Wikibench: Community-Driven Data Curation for AI Evaluation on Wikipedia |
| topic | Human-Computer Interaction Artificial Intelligence |
| url | https://arxiv.org/abs/2402.14147 |