HealthBench: Evaluating Large Language Models Towards Improved Human Health
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918019258646528 |
|---|---|
| author | Arora, Rahul K. Wei, Jason Hicks, Rebecca Soskin Bowman, Preston Quiñonero-Candela, Joaquin Tsimpourlas, Foivos Sharman, Michael Shah, Meghan Vallone, Andrea Beutel, Alex Heidecke, Johannes Singhal, Karan |
| author_facet | Arora, Rahul K. Wei, Jason Hicks, Rebecca Soskin Bowman, Preston Quiñonero-Candela, Joaquin Tsimpourlas, Foivos Sharman, Michael Shah, Meghan Vallone, Andrea Beutel, Alex Heidecke, Johannes Singhal, Karan |
| contents | We present HealthBench, an open-source benchmark measuring the performance and safety of large language models in healthcare. HealthBench consists of 5,000 multi-turn conversations between a model and an individual user or healthcare professional. Responses are evaluated using conversation-specific rubrics created by 262 physicians. Unlike previous multiple-choice or short-answer benchmarks, HealthBench enables realistic, open-ended evaluation through 48,562 unique rubric criteria spanning several health contexts (e.g., emergencies, transforming clinical data, global health) and behavioral dimensions (e.g., accuracy, instruction following, communication). HealthBench performance over the last two years reflects steady initial progress (compare GPT-3.5 Turbo's 16% to GPT-4o's 32%) and more rapid recent improvements (o3 scores 60%). Smaller models have especially improved: GPT-4.1 nano outperforms GPT-4o and is 25 times cheaper. We additionally release two HealthBench variations: HealthBench Consensus, which includes 34 particularly important dimensions of model behavior validated via physician consensus, and HealthBench Hard, where the current top score is 32%. We hope that HealthBench grounds progress towards model development and applications that benefit human health. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_08775 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | HealthBench: Evaluating Large Language Models Towards Improved Human Health Arora, Rahul K. Wei, Jason Hicks, Rebecca Soskin Bowman, Preston Quiñonero-Candela, Joaquin Tsimpourlas, Foivos Sharman, Michael Shah, Meghan Vallone, Andrea Beutel, Alex Heidecke, Johannes Singhal, Karan Computation and Language We present HealthBench, an open-source benchmark measuring the performance and safety of large language models in healthcare. HealthBench consists of 5,000 multi-turn conversations between a model and an individual user or healthcare professional. Responses are evaluated using conversation-specific rubrics created by 262 physicians. Unlike previous multiple-choice or short-answer benchmarks, HealthBench enables realistic, open-ended evaluation through 48,562 unique rubric criteria spanning several health contexts (e.g., emergencies, transforming clinical data, global health) and behavioral dimensions (e.g., accuracy, instruction following, communication). HealthBench performance over the last two years reflects steady initial progress (compare GPT-3.5 Turbo's 16% to GPT-4o's 32%) and more rapid recent improvements (o3 scores 60%). Smaller models have especially improved: GPT-4.1 nano outperforms GPT-4o and is 25 times cheaper. We additionally release two HealthBench variations: HealthBench Consensus, which includes 34 particularly important dimensions of model behavior validated via physician consensus, and HealthBench Hard, where the current top score is 32%. We hope that HealthBench grounds progress towards model development and applications that benefit human health. |
| title | HealthBench: Evaluating Large Language Models Towards Improved Human Health |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.08775 |