Adaptive Crowdsourcing Via Self-Supervised Learning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911769472008192 |
|---|---|
| author | Kagrecha, Anmol Marklund, Henrik Van Roy, Benjamin Jeon, Hong Jun Zeckhauser, Richard |
| author_facet | Kagrecha, Anmol Marklund, Henrik Van Roy, Benjamin Jeon, Hong Jun Zeckhauser, Richard |
| contents | Common crowdsourcing systems average estimates of a latent quantity of interest provided by many crowdworkers to produce a group estimate. We develop a new approach -- predict-each-worker -- that leverages self-supervised learning and a novel aggregation scheme. This approach adapts weights assigned to crowdworkers based on estimates they provided for previous quantities. When skills vary across crowdworkers or their estimates correlate, the weighted sum offers a more accurate group estimate than the average. Existing algorithms such as expectation maximization can, at least in principle, produce similarly accurate group estimates. However, their computational requirements become onerous when complex models, such as neural networks, are required to express relationships among crowdworkers. Predict-each-worker accommodates such complexity as well as many other practical challenges. We analyze the efficacy of predict-each-worker through theoretical and computational studies. Among other things, we establish asymptotic optimality as the number of engagements per crowdworker grows. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_13239 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Adaptive Crowdsourcing Via Self-Supervised Learning Kagrecha, Anmol Marklund, Henrik Van Roy, Benjamin Jeon, Hong Jun Zeckhauser, Richard Machine Learning Human-Computer Interaction Common crowdsourcing systems average estimates of a latent quantity of interest provided by many crowdworkers to produce a group estimate. We develop a new approach -- predict-each-worker -- that leverages self-supervised learning and a novel aggregation scheme. This approach adapts weights assigned to crowdworkers based on estimates they provided for previous quantities. When skills vary across crowdworkers or their estimates correlate, the weighted sum offers a more accurate group estimate than the average. Existing algorithms such as expectation maximization can, at least in principle, produce similarly accurate group estimates. However, their computational requirements become onerous when complex models, such as neural networks, are required to express relationships among crowdworkers. Predict-each-worker accommodates such complexity as well as many other practical challenges. We analyze the efficacy of predict-each-worker through theoretical and computational studies. Among other things, we establish asymptotic optimality as the number of engagements per crowdworker grows. |
| title | Adaptive Crowdsourcing Via Self-Supervised Learning |
| topic | Machine Learning Human-Computer Interaction |
| url | https://arxiv.org/abs/2401.13239 |