MedVAL: Toward Expert-Level Medical Text Validation with Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915780774330368 |
|---|---|
| author | Aali, Asad Bikia, Vasiliki Varma, Maya Chiou, Nicole Ostmeier, Sophie Singhvi, Arnav Paschali, Magdalini Kumar, Ashwin Johnston, Andrew Amador-Martinez, Karimar Guerrero, Eduardo Juan Perez Rivera, Paola Naovi Cruz Gatidis, Sergios Bluethgen, Christian Reis, Eduardo Pontes van Rilland, Eddy D. Zandee Hosamani, Poonam Laxmappa Keet, Kevin R Go, Minjoung Ling, Evelyn Larson, David B. Langlotz, Curtis Daneshjou, Roxana Hom, Jason Koyejo, Sanmi Alsentzer, Emily Chaudhari, Akshay S. |
| author_facet | Aali, Asad Bikia, Vasiliki Varma, Maya Chiou, Nicole Ostmeier, Sophie Singhvi, Arnav Paschali, Magdalini Kumar, Ashwin Johnston, Andrew Amador-Martinez, Karimar Guerrero, Eduardo Juan Perez Rivera, Paola Naovi Cruz Gatidis, Sergios Bluethgen, Christian Reis, Eduardo Pontes van Rilland, Eddy D. Zandee Hosamani, Poonam Laxmappa Keet, Kevin R Go, Minjoung Ling, Evelyn Larson, David B. Langlotz, Curtis Daneshjou, Roxana Hom, Jason Koyejo, Sanmi Alsentzer, Emily Chaudhari, Akshay S. |
| contents | With the growing use of language models (LMs) in clinical environments, there is an immediate need to evaluate the accuracy and safety of LM-generated medical text. Currently, such evaluation relies solely on manual physician review. However, detecting errors in LM-generated text is challenging because 1) manual review is costly and 2) expert-composed reference outputs are often unavailable in real-world settings. While the "LLM-as-a-judge" paradigm offers scalable evaluation, even frontier LMs can miss subtle but clinically significant errors. We propose MedVAL, a novel, self-supervised, data-efficient distillation method that leverages synthetic data to train evaluator LMs to assess whether LM-generated medical outputs are factually consistent with inputs, without requiring physician labels or reference outputs. To evaluate LM performance, we introduce MedVAL-Bench, a dataset of 840 physician-annotated outputs across 6 diverse medical tasks capturing real-world challenges. Across 10 state-of-the-art LMs spanning open-source and proprietary models, MedVAL distillation significantly improves (p < 0.001) alignment with physicians across seen and unseen tasks, increasing average F1 scores from 66% to 83%. Despite strong baseline performance, MedVAL improves the best-performing proprietary LM (GPT-4o) by 8% without training on physician-labeled data, demonstrating a performance statistically non-inferior to a single human expert on a subset annotated by multiple physicians (p < 0.001). To support a scalable, risk-aware pathway towards clinical integration, we open-source: 1) Codebase (https://github.com/StanfordMIMI/MedVAL), 2) MedVAL-Bench (https://huggingface.co/datasets/stanfordmimi/MedVAL-Bench), 3) MedVAL-4B (https://huggingface.co/stanfordmimi/MedVAL-4B). Our benchmark provides evidence of LMs approaching expert-level ability in validating AI-generated medical text. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_03152 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MedVAL: Toward Expert-Level Medical Text Validation with Language Models Aali, Asad Bikia, Vasiliki Varma, Maya Chiou, Nicole Ostmeier, Sophie Singhvi, Arnav Paschali, Magdalini Kumar, Ashwin Johnston, Andrew Amador-Martinez, Karimar Guerrero, Eduardo Juan Perez Rivera, Paola Naovi Cruz Gatidis, Sergios Bluethgen, Christian Reis, Eduardo Pontes van Rilland, Eddy D. Zandee Hosamani, Poonam Laxmappa Keet, Kevin R Go, Minjoung Ling, Evelyn Larson, David B. Langlotz, Curtis Daneshjou, Roxana Hom, Jason Koyejo, Sanmi Alsentzer, Emily Chaudhari, Akshay S. Computation and Language Artificial Intelligence Machine Learning With the growing use of language models (LMs) in clinical environments, there is an immediate need to evaluate the accuracy and safety of LM-generated medical text. Currently, such evaluation relies solely on manual physician review. However, detecting errors in LM-generated text is challenging because 1) manual review is costly and 2) expert-composed reference outputs are often unavailable in real-world settings. While the "LLM-as-a-judge" paradigm offers scalable evaluation, even frontier LMs can miss subtle but clinically significant errors. We propose MedVAL, a novel, self-supervised, data-efficient distillation method that leverages synthetic data to train evaluator LMs to assess whether LM-generated medical outputs are factually consistent with inputs, without requiring physician labels or reference outputs. To evaluate LM performance, we introduce MedVAL-Bench, a dataset of 840 physician-annotated outputs across 6 diverse medical tasks capturing real-world challenges. Across 10 state-of-the-art LMs spanning open-source and proprietary models, MedVAL distillation significantly improves (p < 0.001) alignment with physicians across seen and unseen tasks, increasing average F1 scores from 66% to 83%. Despite strong baseline performance, MedVAL improves the best-performing proprietary LM (GPT-4o) by 8% without training on physician-labeled data, demonstrating a performance statistically non-inferior to a single human expert on a subset annotated by multiple physicians (p < 0.001). To support a scalable, risk-aware pathway towards clinical integration, we open-source: 1) Codebase (https://github.com/StanfordMIMI/MedVAL), 2) MedVAL-Bench (https://huggingface.co/datasets/stanfordmimi/MedVAL-Bench), 3) MedVAL-4B (https://huggingface.co/stanfordmimi/MedVAL-4B). Our benchmark provides evidence of LMs approaching expert-level ability in validating AI-generated medical text. |
| title | MedVAL: Toward Expert-Level Medical Text Validation with Language Models |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2507.03152 |