Comparing Zealous and Restrained AI Recommendations in a Real-World Human-AI Collaboration Task
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914974122639360 |
|---|---|
| author | Xu, Chengyuan Lien, Kuo-Chin Höllerer, Tobias |
| author_facet | Xu, Chengyuan Lien, Kuo-Chin Höllerer, Tobias |
| contents | When designing an AI-assisted decision-making system, there is often a tradeoff between precision and recall in the AI's recommendations. We argue that careful exploitation of this tradeoff can harness the complementary strengths in the human-AI collaboration to significantly improve team performance. We investigate a real-world video anonymization task for which recall is paramount and more costly to improve. We analyze the performance of 78 professional annotators working with a) no AI assistance, b) a high-precision "restrained" AI, and c) a high-recall "zealous" AI in over 3,466 person-hours of annotation work. In comparison, the zealous AI helps human teammates achieve significantly shorter task completion time and higher recall. In a follow-up study, we remove AI assistance for everyone and find negative training effects on annotators trained with the restrained AI. These findings and our analysis point to important implications for the design of AI assistance in recall-demanding scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_11860 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Comparing Zealous and Restrained AI Recommendations in a Real-World Human-AI Collaboration Task Xu, Chengyuan Lien, Kuo-Chin Höllerer, Tobias Human-Computer Interaction Artificial Intelligence Computer Vision and Pattern Recognition H.5.0; I.2.0 When designing an AI-assisted decision-making system, there is often a tradeoff between precision and recall in the AI's recommendations. We argue that careful exploitation of this tradeoff can harness the complementary strengths in the human-AI collaboration to significantly improve team performance. We investigate a real-world video anonymization task for which recall is paramount and more costly to improve. We analyze the performance of 78 professional annotators working with a) no AI assistance, b) a high-precision "restrained" AI, and c) a high-recall "zealous" AI in over 3,466 person-hours of annotation work. In comparison, the zealous AI helps human teammates achieve significantly shorter task completion time and higher recall. In a follow-up study, we remove AI assistance for everyone and find negative training effects on annotators trained with the restrained AI. These findings and our analysis point to important implications for the design of AI assistance in recall-demanding scenarios. |
| title | Comparing Zealous and Restrained AI Recommendations in a Real-World Human-AI Collaboration Task |
| topic | Human-Computer Interaction Artificial Intelligence Computer Vision and Pattern Recognition H.5.0; I.2.0 |
| url | https://arxiv.org/abs/2410.11860 |