Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2507.07988 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908444994306048 |
|---|---|
| author | Zhou, Shuang Xie, Wenya Li, Jiaxi Zhan, Zaifu Song, Meijia Yang, Han Espinoza, Cheyenna Welton, Lindsay Mai, Xinnie Jin, Yanwei Xu, Zidu Chung, Yuen-Hei Xing, Yiyun Tsai, Meng-Han Schaffer, Emma Shi, Yucheng Liu, Ninghao Liu, Zirui Zhang, Rui |
| author_facet | Zhou, Shuang Xie, Wenya Li, Jiaxi Zhan, Zaifu Song, Meijia Yang, Han Espinoza, Cheyenna Welton, Lindsay Mai, Xinnie Jin, Yanwei Xu, Zidu Chung, Yuen-Hei Xing, Yiyun Tsai, Meng-Han Schaffer, Emma Shi, Yucheng Liu, Ninghao Liu, Zirui Zhang, Rui |
| contents | As large language models (LLMs) become increasingly integrated into clinical decision-making, ensuring transparent and trustworthy reasoning is essential. However, existing evaluation strategies of LLMs' medical reasoning capability either suffer from unsatisfactory assessment or poor scalability, and a rigorous benchmark remains lacking. To address this, we introduce MedThink-Bench, a benchmark designed for rigorous, explainable, and scalable assessment of LLMs' medical reasoning. MedThink-Bench comprises 500 challenging questions across ten medical domains, each annotated with expert-crafted step-by-step rationales. Building on this, we propose LLM-w-Ref, a novel evaluation framework that leverages fine-grained rationales and LLM-as-a-Judge mechanisms to assess intermediate reasoning with expert-level fidelity while maintaining scalability. Experiments show that LLM-w-Ref exhibits a strong positive correlation with expert judgments. Benchmarking twelve state-of-the-art LLMs, we find that smaller models (e.g., MedGemma-27B) can surpass larger proprietary counterparts (e.g., OpenAI-o3). Overall, MedThink-Bench offers a foundational tool for evaluating LLMs' medical reasoning, advancing their safe and responsible deployment in clinical practice. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_07988 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Automating Expert-Level Medical Reasoning Evaluation of Large Language Models Zhou, Shuang Xie, Wenya Li, Jiaxi Zhan, Zaifu Song, Meijia Yang, Han Espinoza, Cheyenna Welton, Lindsay Mai, Xinnie Jin, Yanwei Xu, Zidu Chung, Yuen-Hei Xing, Yiyun Tsai, Meng-Han Schaffer, Emma Shi, Yucheng Liu, Ninghao Liu, Zirui Zhang, Rui Computation and Language As large language models (LLMs) become increasingly integrated into clinical decision-making, ensuring transparent and trustworthy reasoning is essential. However, existing evaluation strategies of LLMs' medical reasoning capability either suffer from unsatisfactory assessment or poor scalability, and a rigorous benchmark remains lacking. To address this, we introduce MedThink-Bench, a benchmark designed for rigorous, explainable, and scalable assessment of LLMs' medical reasoning. MedThink-Bench comprises 500 challenging questions across ten medical domains, each annotated with expert-crafted step-by-step rationales. Building on this, we propose LLM-w-Ref, a novel evaluation framework that leverages fine-grained rationales and LLM-as-a-Judge mechanisms to assess intermediate reasoning with expert-level fidelity while maintaining scalability. Experiments show that LLM-w-Ref exhibits a strong positive correlation with expert judgments. Benchmarking twelve state-of-the-art LLMs, we find that smaller models (e.g., MedGemma-27B) can surpass larger proprietary counterparts (e.g., OpenAI-o3). Overall, MedThink-Bench offers a foundational tool for evaluating LLMs' medical reasoning, advancing their safe and responsible deployment in clinical practice. |
| title | Automating Expert-Level Medical Reasoning Evaluation of Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2507.07988 |