Learning from Crowds with Crowd-Kit
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2021
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916195538567168 |
|---|---|
| author | Ustalov, Dmitry Pavlichenko, Nikita Tseitlin, Boris |
| author_facet | Ustalov, Dmitry Pavlichenko, Nikita Tseitlin, Boris |
| contents | This paper presents Crowd-Kit, a general-purpose computational quality control toolkit for crowdsourcing. Crowd-Kit provides efficient and convenient implementations of popular quality control algorithms in Python, including methods for truth inference, deep learning from crowds, and data quality estimation. Our toolkit supports multiple modalities of answers and provides dataset loaders and example notebooks for faster prototyping. We extensively evaluated our toolkit on several datasets of different natures, enabling benchmarking computational quality control methods in a uniform, systematic, and reproducible way using the same codebase. We release our code and data under the Apache License 2.0 at https://github.com/Toloka/crowd-kit. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2109_08584 |
| institution | arXiv |
| publishDate | 2021 |
| record_format | arxiv |
| spellingShingle | Learning from Crowds with Crowd-Kit Ustalov, Dmitry Pavlichenko, Nikita Tseitlin, Boris Human-Computer Interaction Software Engineering G.4 This paper presents Crowd-Kit, a general-purpose computational quality control toolkit for crowdsourcing. Crowd-Kit provides efficient and convenient implementations of popular quality control algorithms in Python, including methods for truth inference, deep learning from crowds, and data quality estimation. Our toolkit supports multiple modalities of answers and provides dataset loaders and example notebooks for faster prototyping. We extensively evaluated our toolkit on several datasets of different natures, enabling benchmarking computational quality control methods in a uniform, systematic, and reproducible way using the same codebase. We release our code and data under the Apache License 2.0 at https://github.com/Toloka/crowd-kit. |
| title | Learning from Crowds with Crowd-Kit |
| topic | Human-Computer Interaction Software Engineering G.4 |
| url | https://arxiv.org/abs/2109.08584 |