MedAgentsBench: Benchmarking Thinking Models and Agent Frameworks for Complex Medical Reasoning
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916657743527936 |
|---|---|
| author | Tang, Xiangru Shao, Daniel Sohn, Jiwoong Chen, Jiapeng Zhang, Jiayi Xiang, Jinyu Wu, Fang Zhao, Yilun Wu, Chenglin Shi, Wenqi Cohan, Arman Gerstein, Mark |
| author_facet | Tang, Xiangru Shao, Daniel Sohn, Jiwoong Chen, Jiapeng Zhang, Jiayi Xiang, Jinyu Wu, Fang Zhao, Yilun Wu, Chenglin Shi, Wenqi Cohan, Arman Gerstein, Mark |
| contents | Large Language Models (LLMs) have shown impressive performance on existing medical question-answering benchmarks. This high performance makes it increasingly difficult to meaningfully evaluate and differentiate advanced methods. We present MedAgentsBench, a benchmark that focuses on challenging medical questions requiring multi-step clinical reasoning, diagnosis formulation, and treatment planning-scenarios where current models still struggle despite their strong performance on standard tests. Drawing from seven established medical datasets, our benchmark addresses three key limitations in existing evaluations: (1) the prevalence of straightforward questions where even base models achieve high performance, (2) inconsistent sampling and evaluation protocols across studies, and (3) lack of systematic analysis of the interplay between performance, cost, and inference time. Through experiments with various base models and reasoning methods, we demonstrate that the latest thinking models, DeepSeek R1 and OpenAI o3, exhibit exceptional performance in complex medical reasoning tasks. Additionally, advanced search-based agent methods offer promising performance-to-cost ratios compared to traditional approaches. Our analysis reveals substantial performance gaps between model families on complex questions and identifies optimal model selections for different computational constraints. Our benchmark and evaluation framework are publicly available at https://github.com/gersteinlab/medagents-benchmark. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_07459 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MedAgentsBench: Benchmarking Thinking Models and Agent Frameworks for Complex Medical Reasoning Tang, Xiangru Shao, Daniel Sohn, Jiwoong Chen, Jiapeng Zhang, Jiayi Xiang, Jinyu Wu, Fang Zhao, Yilun Wu, Chenglin Shi, Wenqi Cohan, Arman Gerstein, Mark Computation and Language Artificial Intelligence Large Language Models (LLMs) have shown impressive performance on existing medical question-answering benchmarks. This high performance makes it increasingly difficult to meaningfully evaluate and differentiate advanced methods. We present MedAgentsBench, a benchmark that focuses on challenging medical questions requiring multi-step clinical reasoning, diagnosis formulation, and treatment planning-scenarios where current models still struggle despite their strong performance on standard tests. Drawing from seven established medical datasets, our benchmark addresses three key limitations in existing evaluations: (1) the prevalence of straightforward questions where even base models achieve high performance, (2) inconsistent sampling and evaluation protocols across studies, and (3) lack of systematic analysis of the interplay between performance, cost, and inference time. Through experiments with various base models and reasoning methods, we demonstrate that the latest thinking models, DeepSeek R1 and OpenAI o3, exhibit exceptional performance in complex medical reasoning tasks. Additionally, advanced search-based agent methods offer promising performance-to-cost ratios compared to traditional approaches. Our analysis reveals substantial performance gaps between model families on complex questions and identifies optimal model selections for different computational constraints. Our benchmark and evaluation framework are publicly available at https://github.com/gersteinlab/medagents-benchmark. |
| title | MedAgentsBench: Benchmarking Thinking Models and Agent Frameworks for Complex Medical Reasoning |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2503.07459 |