Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.27470 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917450344300544 |
|---|---|
| author | Hicks, Rebecca Soskin Trofimov, Mikhail Lim, Dominick Arora, Rahul K. Tsimpourlas, Foivos Bowman, Preston Sharman, Michael Tong, Chi Karthik, Kavin Dugar, Arnav Jagadeesh, Akshay Saab, Khaled Heidecke, Johannes Alexander, Ashley Gross, Nate Singhal, Karan |
| author_facet | Hicks, Rebecca Soskin Trofimov, Mikhail Lim, Dominick Arora, Rahul K. Tsimpourlas, Foivos Bowman, Preston Sharman, Michael Tong, Chi Karthik, Kavin Dugar, Arnav Jagadeesh, Akshay Saab, Khaled Heidecke, Johannes Alexander, Ashley Gross, Nate Singhal, Karan |
| contents | Millions of clinicians use ChatGPT to support clinical care, but evaluations of the most common use cases in model-clinician conversations are limited. We introduce HealthBench Professional, an open benchmark for evaluating large language models on real tasks that clinicians bring to ChatGPT in the course of their work. The benchmark is organized around three common use cases central to clinical practice: care consult, writing and documentation, and medical research. Each example includes a physician-authored conversation with ChatGPT for Clinicians and is scored via rubrics written and iteratively adjudicated by three or more physicians across three phases. HealthBench Professional examples were carefully selected for quality, representativeness, and difficulty for OpenAI's current frontier models, to enable continued measurement of progress. Difficult examples for recent OpenAI models were enriched by roughly 3.5 times relative to the candidate pool of 15,079 examples. Additionally, about one-third of examples involve physicians conducting deliberate adversarial testing of models. As a strong baseline, we also collected human physician responses for all tasks (unbounded time, specialist-matched, web access). The best scoring system, GPT-5.4 in ChatGPT for Clinicians, outperforms base GPT-5.4, all other models, and human physicians. We hope HealthBench Professional provides the healthcare AI community a measure to track frontier model progress in real-world clinical tasks and build systems that clinicians can trust to improve care. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_27470 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats Hicks, Rebecca Soskin Trofimov, Mikhail Lim, Dominick Arora, Rahul K. Tsimpourlas, Foivos Bowman, Preston Sharman, Michael Tong, Chi Karthik, Kavin Dugar, Arnav Jagadeesh, Akshay Saab, Khaled Heidecke, Johannes Alexander, Ashley Gross, Nate Singhal, Karan Computation and Language Millions of clinicians use ChatGPT to support clinical care, but evaluations of the most common use cases in model-clinician conversations are limited. We introduce HealthBench Professional, an open benchmark for evaluating large language models on real tasks that clinicians bring to ChatGPT in the course of their work. The benchmark is organized around three common use cases central to clinical practice: care consult, writing and documentation, and medical research. Each example includes a physician-authored conversation with ChatGPT for Clinicians and is scored via rubrics written and iteratively adjudicated by three or more physicians across three phases. HealthBench Professional examples were carefully selected for quality, representativeness, and difficulty for OpenAI's current frontier models, to enable continued measurement of progress. Difficult examples for recent OpenAI models were enriched by roughly 3.5 times relative to the candidate pool of 15,079 examples. Additionally, about one-third of examples involve physicians conducting deliberate adversarial testing of models. As a strong baseline, we also collected human physician responses for all tasks (unbounded time, specialist-matched, web access). The best scoring system, GPT-5.4 in ChatGPT for Clinicians, outperforms base GPT-5.4, all other models, and human physicians. We hope HealthBench Professional provides the healthcare AI community a measure to track frontier model progress in real-world clinical tasks and build systems that clinicians can trust to improve care. |
| title | HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.27470 |