LEME: Open Large Language Models for Ophthalmology with Advanced Reasoning and Clinical Validation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912692052164608 |
|---|---|
| author | Kim, Hyunjae Ai, Xuguang Srinivasan, Sahana Gilson, Aidan Singer, Maxwell B. Pushpanathan, Krithi Xie, Qianqian Park, Jungwoo Applebaum, Serina Yang, Gabriel Dawei Zou, Minjie Chen, David Ziyou Zou, Ke Sarrafpour, Soshian Liu, Ji Yin, Yu Huang, Jimin Nguyen, Quang Ngoc Long, Erping Wan, Peixing Liu, Dianbo Hintz, Richard Zheng, W. Jim Wang, Sophia Y. Ohno-Machado, Lucila Xu, Hua Adelman, Ron A. Del Priore, Luciano V. Tham, Yih-Chung Chen, Qingyu |
| author_facet | Kim, Hyunjae Ai, Xuguang Srinivasan, Sahana Gilson, Aidan Singer, Maxwell B. Pushpanathan, Krithi Xie, Qianqian Park, Jungwoo Applebaum, Serina Yang, Gabriel Dawei Zou, Minjie Chen, David Ziyou Zou, Ke Sarrafpour, Soshian Liu, Ji Yin, Yu Huang, Jimin Nguyen, Quang Ngoc Long, Erping Wan, Peixing Liu, Dianbo Hintz, Richard Zheng, W. Jim Wang, Sophia Y. Ohno-Machado, Lucila Xu, Hua Adelman, Ron A. Del Priore, Luciano V. Tham, Yih-Chung Chen, Qingyu |
| contents | The rising prevalence of eye diseases poses a growing public health burden. Large language models (LLMs) offer a promising path to reduce documentation workload and support clinical decision-making. However, few have been tailored for ophthalmology, and most evaluations focus mainly on knowledge-based QA without clinically relevant benchmarks or real-world validation. Here, we present LEME, a suite of open-weight LLMs developed through a two-stage process: (1) instruction tuning on 200,000 samples from clinical guidelines, textbooks, and case reports to enhance reasoning and task-following, and (2) reinforcement learning with ~30,000 preference labels to enhance accuracy and informativeness. LEME was evaluated on five curated zero-shot benchmarks spanning tasks such as patient QA, consultation, and treatment planning. It outperformed all seven baselines (all p < 0.004), exceeding GPT-4o by 3.32% (absolute ROUGE-L gain). It was further evaluated on three downstream tasks using deidentified patient data, reviewed by clinicians. In patient QA, LEME received the highest ratings from attending clinicians in 3 out of 4 criteria, with scores of 4.67 for factuality, 4.77 for specificity, 4.79 for completeness, and 4.88 for safety (1-5 scale). Its completeness score surpassed that of expert-written answers (4.79 vs. 4.56; p = 0.015). In visual acuity extraction, LEME achieved the highest F1, outperforming LLaMA-3 by 14.1% and Eye-LLaMA by 59.0%. In a pilot evaluation on assessment and treatment planning for diabetic retinopathy, AMD, and glaucoma, LEME received scores of 4.36 for factuality, 4.55 for specificity, 4.42 for completeness, and 4.36 for safety, approaching attending-level performance. All models, data, and code will be released to support further development and clinical translation, laying the groundwork for improved efficiency and patient care |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_03740 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LEME: Open Large Language Models for Ophthalmology with Advanced Reasoning and Clinical Validation Kim, Hyunjae Ai, Xuguang Srinivasan, Sahana Gilson, Aidan Singer, Maxwell B. Pushpanathan, Krithi Xie, Qianqian Park, Jungwoo Applebaum, Serina Yang, Gabriel Dawei Zou, Minjie Chen, David Ziyou Zou, Ke Sarrafpour, Soshian Liu, Ji Yin, Yu Huang, Jimin Nguyen, Quang Ngoc Long, Erping Wan, Peixing Liu, Dianbo Hintz, Richard Zheng, W. Jim Wang, Sophia Y. Ohno-Machado, Lucila Xu, Hua Adelman, Ron A. Del Priore, Luciano V. Tham, Yih-Chung Chen, Qingyu Computation and Language The rising prevalence of eye diseases poses a growing public health burden. Large language models (LLMs) offer a promising path to reduce documentation workload and support clinical decision-making. However, few have been tailored for ophthalmology, and most evaluations focus mainly on knowledge-based QA without clinically relevant benchmarks or real-world validation. Here, we present LEME, a suite of open-weight LLMs developed through a two-stage process: (1) instruction tuning on 200,000 samples from clinical guidelines, textbooks, and case reports to enhance reasoning and task-following, and (2) reinforcement learning with ~30,000 preference labels to enhance accuracy and informativeness. LEME was evaluated on five curated zero-shot benchmarks spanning tasks such as patient QA, consultation, and treatment planning. It outperformed all seven baselines (all p < 0.004), exceeding GPT-4o by 3.32% (absolute ROUGE-L gain). It was further evaluated on three downstream tasks using deidentified patient data, reviewed by clinicians. In patient QA, LEME received the highest ratings from attending clinicians in 3 out of 4 criteria, with scores of 4.67 for factuality, 4.77 for specificity, 4.79 for completeness, and 4.88 for safety (1-5 scale). Its completeness score surpassed that of expert-written answers (4.79 vs. 4.56; p = 0.015). In visual acuity extraction, LEME achieved the highest F1, outperforming LLaMA-3 by 14.1% and Eye-LLaMA by 59.0%. In a pilot evaluation on assessment and treatment planning for diabetic retinopathy, AMD, and glaucoma, LEME received scores of 4.36 for factuality, 4.55 for specificity, 4.42 for completeness, and 4.36 for safety, approaching attending-level performance. All models, data, and code will be released to support further development and clinical translation, laying the groundwork for improved efficiency and patient care |
| title | LEME: Open Large Language Models for Ophthalmology with Advanced Reasoning and Clinical Validation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.03740 |