Answer, Refuse, or Guess? Investigating Risk-Aware Decision Making in Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911184479846400 |
|---|---|
| author | Wu, Cheng-Kuang Tam, Zhi Rui Lin, Chieh-Yen Chen, Yun-Nung Lee, Hung-yi |
| author_facet | Wu, Cheng-Kuang Tam, Zhi Rui Lin, Chieh-Yen Chen, Yun-Nung Lee, Hung-yi |
| contents | Language models (LMs) are increasingly used to build agents that can act autonomously to achieve goals. During this automatic process, agents need to take a series of actions, some of which might lead to severe consequences if incorrect actions are taken. Therefore, such agents must sometimes defer-refusing to act when their confidence is insufficient-to avoid the potential cost of incorrect actions. Because the severity of consequences varies across applications, the tendency to defer should also vary: in low-risk settings agents should answer more freely, while in high-risk settings their decisions should be more conservative. We study this "answer-or-defer" problem with an evaluation framework that systematically varies human-specified risk structures-rewards and penalties for correct answers, incorrect answers, and refusals $(r_{\mathrm{cor}},r_{\mathrm{inc}}, r_{\mathrm{ref}})$-while keeping tasks fixed. This design evaluates LMs' risk-aware decision policies by measuring their ability to maximize expected reward. Across multiple datasets and models, we identify flaws in their decision policies: LMs tend to over-answer in high-risk settings and over-defer in low-risk settings. After analyzing the potential cause of such flaws, we find that a simple skill-decomposition method, which isolates the independent skills required for answer-or-defer decision making, can consistently improve LMs' decision policies. Our results highlight the current limitations of LMs in risk-conditioned decision making and provide practical guidance for deploying more reliable LM-based agents across applications of varying risk levels. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01332 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Answer, Refuse, or Guess? Investigating Risk-Aware Decision Making in Language Models Wu, Cheng-Kuang Tam, Zhi Rui Lin, Chieh-Yen Chen, Yun-Nung Lee, Hung-yi Computation and Language Artificial Intelligence Language models (LMs) are increasingly used to build agents that can act autonomously to achieve goals. During this automatic process, agents need to take a series of actions, some of which might lead to severe consequences if incorrect actions are taken. Therefore, such agents must sometimes defer-refusing to act when their confidence is insufficient-to avoid the potential cost of incorrect actions. Because the severity of consequences varies across applications, the tendency to defer should also vary: in low-risk settings agents should answer more freely, while in high-risk settings their decisions should be more conservative. We study this "answer-or-defer" problem with an evaluation framework that systematically varies human-specified risk structures-rewards and penalties for correct answers, incorrect answers, and refusals $(r_{\mathrm{cor}},r_{\mathrm{inc}}, r_{\mathrm{ref}})$-while keeping tasks fixed. This design evaluates LMs' risk-aware decision policies by measuring their ability to maximize expected reward. Across multiple datasets and models, we identify flaws in their decision policies: LMs tend to over-answer in high-risk settings and over-defer in low-risk settings. After analyzing the potential cause of such flaws, we find that a simple skill-decomposition method, which isolates the independent skills required for answer-or-defer decision making, can consistently improve LMs' decision policies. Our results highlight the current limitations of LMs in risk-conditioned decision making and provide practical guidance for deploying more reliable LM-based agents across applications of varying risk levels. |
| title | Answer, Refuse, or Guess? Investigating Risk-Aware Decision Making in Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2503.01332 |