MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applications
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912846915305472 |
|---|---|
| author | Kanithi, Praveenkumar Christophe, Clément Pimentel, Marco AF Raha, Tathagata Munjal, Prateek Saadi, Nada Javed, Hamza A Maslenkova, Svetlana Hayat, Nasir Rajan, Ronnie Khan, Shadab |
| author_facet | Kanithi, Praveenkumar Christophe, Clément Pimentel, Marco AF Raha, Tathagata Munjal, Prateek Saadi, Nada Javed, Hamza A Maslenkova, Svetlana Hayat, Nasir Rajan, Ronnie Khan, Shadab |
| contents | While Large Language Models (LLMs) achieve superhuman performance on standardized medical licensing exams, these static benchmarks have become saturated and increasingly disconnected from the functional requirements of clinical workflows. To bridge the gap between theoretical capability and verified utility, we introduce MEDIC, a comprehensive evaluation framework establishing leading indicators across various clinical dimensions. Beyond standard question-answering, we assess operational capabilities using deterministic execution protocols and a novel Cross-Examination Framework (CEF), which quantifies information fidelity and hallucination rates without reliance on reference texts. Our evaluation across a heterogeneous task suite exposes critical performance trade-offs: we identify a significant knowledge-execution gap, where proficiency in static retrieval does not predict success in operational tasks such as clinical calculation or SQL generation. Furthermore, we observe a divergence between passive safety (refusal) and active safety (error detection), revealing that models fine-tuned for high refusal rates often fail to reliably audit clinical documentation for factual accuracy. These findings demonstrate that no single architecture dominates across all dimensions, highlighting the necessity of a portfolio approach to clinical model deployment. As part of this investigation, we released a public leaderboard on Hugging Face.\footnote{https://huggingface.co/spaces/m42-health/MEDIC-Benchmark} |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_07314 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applications Kanithi, Praveenkumar Christophe, Clément Pimentel, Marco AF Raha, Tathagata Munjal, Prateek Saadi, Nada Javed, Hamza A Maslenkova, Svetlana Hayat, Nasir Rajan, Ronnie Khan, Shadab Computation and Language Artificial Intelligence While Large Language Models (LLMs) achieve superhuman performance on standardized medical licensing exams, these static benchmarks have become saturated and increasingly disconnected from the functional requirements of clinical workflows. To bridge the gap between theoretical capability and verified utility, we introduce MEDIC, a comprehensive evaluation framework establishing leading indicators across various clinical dimensions. Beyond standard question-answering, we assess operational capabilities using deterministic execution protocols and a novel Cross-Examination Framework (CEF), which quantifies information fidelity and hallucination rates without reliance on reference texts. Our evaluation across a heterogeneous task suite exposes critical performance trade-offs: we identify a significant knowledge-execution gap, where proficiency in static retrieval does not predict success in operational tasks such as clinical calculation or SQL generation. Furthermore, we observe a divergence between passive safety (refusal) and active safety (error detection), revealing that models fine-tuned for high refusal rates often fail to reliably audit clinical documentation for factual accuracy. These findings demonstrate that no single architecture dominates across all dimensions, highlighting the necessity of a portfolio approach to clinical model deployment. As part of this investigation, we released a public leaderboard on Hugging Face.\footnote{https://huggingface.co/spaces/m42-health/MEDIC-Benchmark} |
| title | MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applications |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2409.07314 |