Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.16679 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910234056851456 |
|---|---|
| author | Chen, Haolin Metelski, Deon Qi, Leon Xia, Tao Lee, Joonyul Brown, Steve Riley, Kevin Wang, Frank Liu, T. Y. Alvin MD, Hank Capps Tang, Zeyu Song, Xiangchen Kong, Lingjing Feng, Fan Zeng, Tianyi Liu, Zhiwei Ma, Zixian Jiang, Hang Geng, Fangli Yuan, Yuan You, Chenyu Wen, Qingsong Wei, Hua Fu, Yanjie Zhao, Yue Yang, Carl Huang, Biwei Zhang, Kun Xiong, Caiming Koyejo, Sanmi Xing, Eric P. Yu, Philip S. Yao, Weiran |
| author_facet | Chen, Haolin Metelski, Deon Qi, Leon Xia, Tao Lee, Joonyul Brown, Steve Riley, Kevin Wang, Frank Liu, T. Y. Alvin MD, Hank Capps Tang, Zeyu Song, Xiangchen Kong, Lingjing Feng, Fan Zeng, Tianyi Liu, Zhiwei Ma, Zixian Jiang, Hang Geng, Fangli Yuan, Yuan You, Chenyu Wen, Qingsong Wei, Hua Fu, Yanjie Zhao, Yue Yang, Carl Huang, Biwei Zhang, Kun Xiong, Caiming Koyejo, Sanmi Xing, Eric P. Yu, Philip S. Yao, Weiran |
| contents | End-to-end automation of realistic healthcare operations stresses three capabilities underrepresented in current benchmarks: policy density, decisions must be grounded in a large library of medical, insurance, and operational rules; Multi-role composition: a single task requires the agent to play multiple roles with handoffs; and multilateral interaction: intermediate workflow steps are multi-turn dialogs, such as peer-to-peer review and patient outreach. We introduce $χ$-Bench, a benchmark of long-horizon healthcare workflows across three domains: provider prior authorization, payer utilization management, and care management. Each task hands the agent a clinical case in a high-fidelity simulator of 20 healthcare apps exposed via 87 MCP tools, which it must drive to a terminal status through tool calls and writing the role's artifacts, guided by a 1,290+ document managed-care operations handbook skill. Across 30 agent harness/models configurations, the best agent resolves only 28.0% of tasks, no agent clears 20% on strict pass^3, and executing all tasks in a single session slumps the performance to 3.8%. These results raise the hypothesis that similar gaps are likely to surface in other policy-dense, role-composed, irreversible enterprise domains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_16679 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows? Chen, Haolin Metelski, Deon Qi, Leon Xia, Tao Lee, Joonyul Brown, Steve Riley, Kevin Wang, Frank Liu, T. Y. Alvin MD, Hank Capps Tang, Zeyu Song, Xiangchen Kong, Lingjing Feng, Fan Zeng, Tianyi Liu, Zhiwei Ma, Zixian Jiang, Hang Geng, Fangli Yuan, Yuan You, Chenyu Wen, Qingsong Wei, Hua Fu, Yanjie Zhao, Yue Yang, Carl Huang, Biwei Zhang, Kun Xiong, Caiming Koyejo, Sanmi Xing, Eric P. Yu, Philip S. Yao, Weiran Computation and Language Artificial Intelligence End-to-end automation of realistic healthcare operations stresses three capabilities underrepresented in current benchmarks: policy density, decisions must be grounded in a large library of medical, insurance, and operational rules; Multi-role composition: a single task requires the agent to play multiple roles with handoffs; and multilateral interaction: intermediate workflow steps are multi-turn dialogs, such as peer-to-peer review and patient outreach. We introduce $χ$-Bench, a benchmark of long-horizon healthcare workflows across three domains: provider prior authorization, payer utilization management, and care management. Each task hands the agent a clinical case in a high-fidelity simulator of 20 healthcare apps exposed via 87 MCP tools, which it must drive to a terminal status through tool calls and writing the role's artifacts, guided by a 1,290+ document managed-care operations handbook skill. Across 30 agent harness/models configurations, the best agent resolves only 28.0% of tasks, no agent clears 20% on strict pass^3, and executing all tasks in a single session slumps the performance to 3.8%. These results raise the hypothesis that similar gaps are likely to surface in other policy-dense, role-composed, irreversible enterprise domains. |
| title | CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows? |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2605.16679 |