LLMs as Workers in Human-Computational Algorithms? Replicating Crowdsourcing Pipelines with LLMs
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912180512751616 |
|---|---|
| author | Wu, Tongshuang Zhu, Haiyi Albayrak, Maya Axon, Alexis Bertsch, Amanda Deng, Wenxing Ding, Ziqi Guo, Bill Gururaja, Sireesh Kuo, Tzu-Sheng Liang, Jenny T. Liu, Ryan Mandal, Ihita Milbauer, Jeremiah Ni, Xiaolin Padmanabhan, Namrata Ramkumar, Subhashini Sudjianto, Alexis Taylor, Jordan Tseng, Ying-Jui Vaidos, Patricia Wu, Zhijin Wu, Wei Yang, Chenyang |
| author_facet | Wu, Tongshuang Zhu, Haiyi Albayrak, Maya Axon, Alexis Bertsch, Amanda Deng, Wenxing Ding, Ziqi Guo, Bill Gururaja, Sireesh Kuo, Tzu-Sheng Liang, Jenny T. Liu, Ryan Mandal, Ihita Milbauer, Jeremiah Ni, Xiaolin Padmanabhan, Namrata Ramkumar, Subhashini Sudjianto, Alexis Taylor, Jordan Tseng, Ying-Jui Vaidos, Patricia Wu, Zhijin Wu, Wei Yang, Chenyang |
| contents | LLMs have shown promise in replicating human-like behavior in crowdsourcing tasks that were previously thought to be exclusive to human abilities. However, current efforts focus mainly on simple atomic tasks. We explore whether LLMs can replicate more complex crowdsourcing pipelines. We find that modern LLMs can simulate some of crowdworkers' abilities in these ``human computation algorithms,'' but the level of success is variable and influenced by requesters' understanding of LLM capabilities, the specific skills required for sub-tasks, and the optimal interaction modality for performing these sub-tasks. We reflect on human and LLMs' different sensitivities to instructions, stress the importance of enabling human-facing safeguards for LLMs, and discuss the potential of training humans and LLMs with complementary skill sets. Crucially, we show that replicating crowdsourcing pipelines offers a valuable platform to investigate 1) the relative LLM strengths on different tasks (by cross-comparing their performances on sub-tasks) and 2) LLMs' potential in complex tasks, where they can complete part of the tasks while leaving others to humans. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2307_10168 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | LLMs as Workers in Human-Computational Algorithms? Replicating Crowdsourcing Pipelines with LLMs Wu, Tongshuang Zhu, Haiyi Albayrak, Maya Axon, Alexis Bertsch, Amanda Deng, Wenxing Ding, Ziqi Guo, Bill Gururaja, Sireesh Kuo, Tzu-Sheng Liang, Jenny T. Liu, Ryan Mandal, Ihita Milbauer, Jeremiah Ni, Xiaolin Padmanabhan, Namrata Ramkumar, Subhashini Sudjianto, Alexis Taylor, Jordan Tseng, Ying-Jui Vaidos, Patricia Wu, Zhijin Wu, Wei Yang, Chenyang Computation and Language Human-Computer Interaction LLMs have shown promise in replicating human-like behavior in crowdsourcing tasks that were previously thought to be exclusive to human abilities. However, current efforts focus mainly on simple atomic tasks. We explore whether LLMs can replicate more complex crowdsourcing pipelines. We find that modern LLMs can simulate some of crowdworkers' abilities in these ``human computation algorithms,'' but the level of success is variable and influenced by requesters' understanding of LLM capabilities, the specific skills required for sub-tasks, and the optimal interaction modality for performing these sub-tasks. We reflect on human and LLMs' different sensitivities to instructions, stress the importance of enabling human-facing safeguards for LLMs, and discuss the potential of training humans and LLMs with complementary skill sets. Crucially, we show that replicating crowdsourcing pipelines offers a valuable platform to investigate 1) the relative LLM strengths on different tasks (by cross-comparing their performances on sub-tasks) and 2) LLMs' potential in complex tasks, where they can complete part of the tasks while leaving others to humans. |
| title | LLMs as Workers in Human-Computational Algorithms? Replicating Crowdsourcing Pipelines with LLMs |
| topic | Computation and Language Human-Computer Interaction |
| url | https://arxiv.org/abs/2307.10168 |