MedBrowseComp: Benchmarking Medical Deep Research and Computer Use
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909617928273920 |
|---|---|
| author | Chen, Shan Moreira, Pedro Xiao, Yuxin Schmidgall, Sam Warner, Jeremy Aerts, Hugo Hartvigsen, Thomas Gallifant, Jack Bitterman, Danielle S. |
| author_facet | Chen, Shan Moreira, Pedro Xiao, Yuxin Schmidgall, Sam Warner, Jeremy Aerts, Hugo Hartvigsen, Thomas Gallifant, Jack Bitterman, Danielle S. |
| contents | Large language models (LLMs) are increasingly envisioned as decision-support tools in clinical practice, yet safe clinical reasoning demands integrating heterogeneous knowledge bases -- trials, primary studies, regulatory documents, and cost data -- under strict accuracy constraints. Existing evaluations often rely on synthetic prompts, reduce the task to single-hop factoid queries, or conflate reasoning with open-ended generation, leaving their real-world utility unclear. To close this gap, we present MedBrowseComp, the first benchmark that systematically tests an agent's ability to reliably retrieve and synthesize multi-hop medical facts from live, domain-specific knowledge bases. MedBrowseComp contains more than 1,000 human-curated questions that mirror clinical scenarios where practitioners must reconcile fragmented or conflicting information to reach an up-to-date conclusion. Applying MedBrowseComp to frontier agentic systems reveals performance shortfalls as low as ten percent, exposing a critical gap between current LLM capabilities and the rigor demanded in clinical settings. MedBrowseComp therefore offers a clear testbed for reliable medical information seeking and sets concrete goals for future model and toolchain upgrades. You can visit our project page at: https://moreirap12.github.io/mbc-browse-app/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_14963 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MedBrowseComp: Benchmarking Medical Deep Research and Computer Use Chen, Shan Moreira, Pedro Xiao, Yuxin Schmidgall, Sam Warner, Jeremy Aerts, Hugo Hartvigsen, Thomas Gallifant, Jack Bitterman, Danielle S. Computation and Language Large language models (LLMs) are increasingly envisioned as decision-support tools in clinical practice, yet safe clinical reasoning demands integrating heterogeneous knowledge bases -- trials, primary studies, regulatory documents, and cost data -- under strict accuracy constraints. Existing evaluations often rely on synthetic prompts, reduce the task to single-hop factoid queries, or conflate reasoning with open-ended generation, leaving their real-world utility unclear. To close this gap, we present MedBrowseComp, the first benchmark that systematically tests an agent's ability to reliably retrieve and synthesize multi-hop medical facts from live, domain-specific knowledge bases. MedBrowseComp contains more than 1,000 human-curated questions that mirror clinical scenarios where practitioners must reconcile fragmented or conflicting information to reach an up-to-date conclusion. Applying MedBrowseComp to frontier agentic systems reveals performance shortfalls as low as ten percent, exposing a critical gap between current LLM capabilities and the rigor demanded in clinical settings. MedBrowseComp therefore offers a clear testbed for reliable medical information seeking and sets concrete goals for future model and toolchain upgrades. You can visit our project page at: https://moreirap12.github.io/mbc-browse-app/ |
| title | MedBrowseComp: Benchmarking Medical Deep Research and Computer Use |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.14963 |