Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.26136 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915754446684160 |
|---|---|
| author | Grundmann, Paul Fast, Dennis Frick, Jan Steffek, Thomas Gers, Felix Nejdl, Wolfgang Löser, Alexander |
| author_facet | Grundmann, Paul Fast, Dennis Frick, Jan Steffek, Thomas Gers, Felix Nejdl, Wolfgang Löser, Alexander |
| contents | With their growing capabilities, generative large language models (LLMs) are being increasingly investigated for complex medical tasks. However, their effectiveness in real-world clinical applications remains underexplored. To address this, we present CliniBench, the first benchmark that enables comparability of well-studied encoder-based classifiers and generative LLMs for discharge diagnosis prediction from admission notes in MIMIC-IV dataset. Our extensive study compares 12 generative LLMs and 3 encoder-based classifiers and demonstrates that encoder-based classifiers consistently outperform generative models in diagnosis prediction. We assess several retrieval augmentation strategies for in-context learning from similar patients and find that they provide notable performance improvements for generative LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_26136 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CliniBench: A Clinical Outcome Prediction Benchmark for Generative and Encoder-Based Language Models Grundmann, Paul Fast, Dennis Frick, Jan Steffek, Thomas Gers, Felix Nejdl, Wolfgang Löser, Alexander Computation and Language With their growing capabilities, generative large language models (LLMs) are being increasingly investigated for complex medical tasks. However, their effectiveness in real-world clinical applications remains underexplored. To address this, we present CliniBench, the first benchmark that enables comparability of well-studied encoder-based classifiers and generative LLMs for discharge diagnosis prediction from admission notes in MIMIC-IV dataset. Our extensive study compares 12 generative LLMs and 3 encoder-based classifiers and demonstrates that encoder-based classifiers consistently outperform generative models in diagnosis prediction. We assess several retrieval augmentation strategies for in-context learning from similar patients and find that they provide notable performance improvements for generative LLMs. |
| title | CliniBench: A Clinical Outcome Prediction Benchmark for Generative and Encoder-Based Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2509.26136 |