Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2504.19467 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918413693091840 |
|---|---|
| author | Wu, Jiageng Gu, Bowen Zhou, Ren Xie, Kevin Snyder, Doug Jiang, Yixing Carducci, Valentina Wyss, Richard Desai, Rishi J Alsentzer, Emily Celi, Leo Anthony Rodman, Adam Schneeweiss, Sebastian Chen, Jonathan H. Romero-Brufau, Santiago Lin, Kueiyu Joshua Yang, Jie |
| author_facet | Wu, Jiageng Gu, Bowen Zhou, Ren Xie, Kevin Snyder, Doug Jiang, Yixing Carducci, Valentina Wyss, Richard Desai, Rishi J Alsentzer, Emily Celi, Leo Anthony Rodman, Adam Schneeweiss, Sebastian Chen, Jonathan H. Romero-Brufau, Santiago Lin, Kueiyu Joshua Yang, Jie |
| contents | Large language models (LLMs) hold great promise for medical applications and are evolving rapidly, with new models being released at an accelerated pace. However, benchmarking on large-scale real-world data such as electronic health records (EHRs) is critical, as clinical decisions are directly informed by these sources, yet current evaluations remain limited. Most existing benchmarks rely on medical exam-style questions or PubMed-derived text, failing to capture the complexity of real-world clinical data. Others focus narrowly on specific application scenarios, limiting their generalizability across broader clinical use. To address this gap, we present BRIDGE, a comprehensive multilingual benchmark comprising 87 tasks sourced from real-world clinical data sources across nine languages. It covers eight major task types spanning the entire continuum of patient care across six clinical stages and 20 representative applications, including triage and referral, consultation, information extraction, diagnosis, prognosis, and billing coding, and involves 14 clinical specialties. We systematically evaluated 95 LLMs (including DeepSeek-R1, GPT-4o, Gemini series, and Qwen3 series) under various inference strategies. Our results reveal substantial performance variation across model sizes, languages, natural language processing tasks, and clinical specialties. Notably, we demonstrate that open-source LLMs can achieve performance comparable to proprietary models, while medically fine-tuned LLMs based on older architectures often underperform versus updated general-purpose models. The BRIDGE and its corresponding leaderboard serve as a foundational resource and a unique reference for the development and evaluation of new LLMs in real-world clinical text understanding.
The BRIDGE leaderboard: https://huggingface.co/spaces/YLab-Open/BRIDGE-Medical-Leaderboard |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_19467 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text Wu, Jiageng Gu, Bowen Zhou, Ren Xie, Kevin Snyder, Doug Jiang, Yixing Carducci, Valentina Wyss, Richard Desai, Rishi J Alsentzer, Emily Celi, Leo Anthony Rodman, Adam Schneeweiss, Sebastian Chen, Jonathan H. Romero-Brufau, Santiago Lin, Kueiyu Joshua Yang, Jie Computation and Language Artificial Intelligence Large language models (LLMs) hold great promise for medical applications and are evolving rapidly, with new models being released at an accelerated pace. However, benchmarking on large-scale real-world data such as electronic health records (EHRs) is critical, as clinical decisions are directly informed by these sources, yet current evaluations remain limited. Most existing benchmarks rely on medical exam-style questions or PubMed-derived text, failing to capture the complexity of real-world clinical data. Others focus narrowly on specific application scenarios, limiting their generalizability across broader clinical use. To address this gap, we present BRIDGE, a comprehensive multilingual benchmark comprising 87 tasks sourced from real-world clinical data sources across nine languages. It covers eight major task types spanning the entire continuum of patient care across six clinical stages and 20 representative applications, including triage and referral, consultation, information extraction, diagnosis, prognosis, and billing coding, and involves 14 clinical specialties. We systematically evaluated 95 LLMs (including DeepSeek-R1, GPT-4o, Gemini series, and Qwen3 series) under various inference strategies. Our results reveal substantial performance variation across model sizes, languages, natural language processing tasks, and clinical specialties. Notably, we demonstrate that open-source LLMs can achieve performance comparable to proprietary models, while medically fine-tuned LLMs based on older architectures often underperform versus updated general-purpose models. The BRIDGE and its corresponding leaderboard serve as a foundational resource and a unique reference for the development and evaluation of new LLMs in real-world clinical text understanding. The BRIDGE leaderboard: https://huggingface.co/spaces/YLab-Open/BRIDGE-Medical-Leaderboard |
| title | BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2504.19467 |