Understanding the Effects of Miscalibrated AI Confidence on User Trust, Reliance, and Decision Efficacy
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908561761632256 |
|---|---|
| author | Li, Jingshu Yang, Yitian Zhang, Renwen Liao, Q. Vera Song, Tianqi Xu, Zhengtao Lee, Yi-chieh |
| author_facet | Li, Jingshu Yang, Yitian Zhang, Renwen Liao, Q. Vera Song, Tianqi Xu, Zhengtao Lee, Yi-chieh |
| contents | Providing well-calibrated AI confidence can help promote users' appropriate trust in and reliance on AI, which are essential for AI-assisted decision-making. However, calibrating AI confidence -- providing confidence score that accurately reflects the true likelihood of AI being correct -- is known to be challenging. To understand the effects of AI confidence miscalibration, we conducted our first experiment. The results indicate that miscalibrated AI confidence impairs users' appropriate reliance and reduces AI-assisted decision-making efficacy, and AI miscalibration is difficult for users to detect. Then, in our second experiment, we examined whether communicating AI confidence calibration levels could mitigate the above issues. We find that it helps users to detect AI miscalibration. Nevertheless, since such communication decreases users' trust in uncalibrated AI, leading to high under-reliance, it does not improve the decision efficacy. We discuss design implications based on these findings and future directions to address risks and ethical concerns associated with AI miscalibration. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_07632 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Understanding the Effects of Miscalibrated AI Confidence on User Trust, Reliance, and Decision Efficacy Li, Jingshu Yang, Yitian Zhang, Renwen Liao, Q. Vera Song, Tianqi Xu, Zhengtao Lee, Yi-chieh Artificial Intelligence Human-Computer Interaction Providing well-calibrated AI confidence can help promote users' appropriate trust in and reliance on AI, which are essential for AI-assisted decision-making. However, calibrating AI confidence -- providing confidence score that accurately reflects the true likelihood of AI being correct -- is known to be challenging. To understand the effects of AI confidence miscalibration, we conducted our first experiment. The results indicate that miscalibrated AI confidence impairs users' appropriate reliance and reduces AI-assisted decision-making efficacy, and AI miscalibration is difficult for users to detect. Then, in our second experiment, we examined whether communicating AI confidence calibration levels could mitigate the above issues. We find that it helps users to detect AI miscalibration. Nevertheless, since such communication decreases users' trust in uncalibrated AI, leading to high under-reliance, it does not improve the decision efficacy. We discuss design implications based on these findings and future directions to address risks and ethical concerns associated with AI miscalibration. |
| title | Understanding the Effects of Miscalibrated AI Confidence on User Trust, Reliance, and Decision Efficacy |
| topic | Artificial Intelligence Human-Computer Interaction |
| url | https://arxiv.org/abs/2402.07632 |