Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911640480382976 |
|---|---|
| author | Warner, Benjamin Grandhi, Ratna Sagari Kieffer, Max Ouraq, Aymane Panigrahi, Saurav Ambwani, Geetu Bagga, Kunal Khandekar, Nikhil Hariharan, Arya Mishra, Nishant Ram, Manish Yang, Shamus Sim Zi Essouaied, Ahmed Moyondafoluwa, Adepoju Jeremiah Scholz, Robert Huang, Bofeng Beavers, Molly Gureja, Srishti Mahishi, Anish Khan, Sameed Griot, Maxime Batra, Hunar Delbrouck, Jean-Benoit Bharadwaj, Siddhant Clark, Ronald Vashist, Ashish Zafar, Anas Murali, Leema Krishna Deshpande, Harsh Patel, Ameen Brown, William Hagemann, Johannes Lane, Connor Scotti, Paul Steven Abraham, Tanishq Mathew |
| author_facet | Warner, Benjamin Grandhi, Ratna Sagari Kieffer, Max Ouraq, Aymane Panigrahi, Saurav Ambwani, Geetu Bagga, Kunal Khandekar, Nikhil Hariharan, Arya Mishra, Nishant Ram, Manish Yang, Shamus Sim Zi Essouaied, Ahmed Moyondafoluwa, Adepoju Jeremiah Scholz, Robert Huang, Bofeng Beavers, Molly Gureja, Srishti Mahishi, Anish Khan, Sameed Griot, Maxime Batra, Hunar Delbrouck, Jean-Benoit Bharadwaj, Siddhant Clark, Ronald Vashist, Ashish Zafar, Anas Murali, Leema Krishna Deshpande, Harsh Patel, Ameen Brown, William Hagemann, Johannes Lane, Connor Scotti, Paul Steven Abraham, Tanishq Mathew |
| contents | Evaluating large language models (LLMs) for medical applications remains challenging due to benchmark saturation, limited data accessibility, and insufficient coverage of relevant tasks. Existing suites have either saturated, heavily depend on restricted datasets, or lack comprehensive model coverage. We introduce Medmarks, a fully open-source evaluation suite with 30 benchmarks spanning question answering, information extraction, medical calculations, and open-ended clinical reasoning. We perform a systematic evaluation of 61 models across 71 configurations using verifiable metrics and LLM-as-a-Judge. Our results show that frontier reasoning models (Gemini 3 Pro Preview, GPT-5.1, & GPT-5.2) achieve the highest performance across both benchmarks, most frontier proprietary models are significantly more token efficient than open-weight alternatives, medically fine-tuned models outperform their generalist counterparts, and that models are susceptible to answer-order bias (particularly smaller models and Grok 4). A subset of our evals (Medmarks-T) can be directly used as reinforcement learning environments to post-train LLMs for medical reasoning. Code is available at https://github.com/MedARC-AI/Medmarks |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_01417 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks Warner, Benjamin Grandhi, Ratna Sagari Kieffer, Max Ouraq, Aymane Panigrahi, Saurav Ambwani, Geetu Bagga, Kunal Khandekar, Nikhil Hariharan, Arya Mishra, Nishant Ram, Manish Yang, Shamus Sim Zi Essouaied, Ahmed Moyondafoluwa, Adepoju Jeremiah Scholz, Robert Huang, Bofeng Beavers, Molly Gureja, Srishti Mahishi, Anish Khan, Sameed Griot, Maxime Batra, Hunar Delbrouck, Jean-Benoit Bharadwaj, Siddhant Clark, Ronald Vashist, Ashish Zafar, Anas Murali, Leema Krishna Deshpande, Harsh Patel, Ameen Brown, William Hagemann, Johannes Lane, Connor Scotti, Paul Steven Abraham, Tanishq Mathew Computation and Language Artificial Intelligence Evaluating large language models (LLMs) for medical applications remains challenging due to benchmark saturation, limited data accessibility, and insufficient coverage of relevant tasks. Existing suites have either saturated, heavily depend on restricted datasets, or lack comprehensive model coverage. We introduce Medmarks, a fully open-source evaluation suite with 30 benchmarks spanning question answering, information extraction, medical calculations, and open-ended clinical reasoning. We perform a systematic evaluation of 61 models across 71 configurations using verifiable metrics and LLM-as-a-Judge. Our results show that frontier reasoning models (Gemini 3 Pro Preview, GPT-5.1, & GPT-5.2) achieve the highest performance across both benchmarks, most frontier proprietary models are significantly more token efficient than open-weight alternatives, medically fine-tuned models outperform their generalist counterparts, and that models are susceptible to answer-order bias (particularly smaller models and Grok 4). A subset of our evals (Medmarks-T) can be directly used as reinforcement learning environments to post-train LLMs for medical reasoning. Code is available at https://github.com/MedARC-AI/Medmarks |
| title | Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2605.01417 |