MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929533651779584 |
|---|---|
| author | Codella, Noel C. F. Jin, Ying Jain, Shrey Gu, Yu Lee, Ho Hin Abacha, Asma Ben Santamaria-Pang, Alberto Guyman, Will Sangani, Naiteek Zhang, Sheng Poon, Hoifung Hyland, Stephanie Bannur, Shruthi Alvarez-Valle, Javier Li, Xue Garrett, John McMillan, Alan Rajguru, Gaurav Maddi, Madhu Vijayrania, Nilesh Bhimai, Rehaan Mecklenburg, Nick Jain, Rupal Holstein, Daniel Gaur, Naveen Aski, Vijay Hwang, Jenq-Neng Lin, Thomas Tarapov, Ivan Lungren, Matthew Wei, Mu |
| author_facet | Codella, Noel C. F. Jin, Ying Jain, Shrey Gu, Yu Lee, Ho Hin Abacha, Asma Ben Santamaria-Pang, Alberto Guyman, Will Sangani, Naiteek Zhang, Sheng Poon, Hoifung Hyland, Stephanie Bannur, Shruthi Alvarez-Valle, Javier Li, Xue Garrett, John McMillan, Alan Rajguru, Gaurav Maddi, Madhu Vijayrania, Nilesh Bhimai, Rehaan Mecklenburg, Nick Jain, Rupal Holstein, Daniel Gaur, Naveen Aski, Vijay Hwang, Jenq-Neng Lin, Thomas Tarapov, Ivan Lungren, Matthew Wei, Mu |
| contents | In this work, we present MedImageInsight, an open-source medical imaging embedding model. MedImageInsight is trained on medical images with associated text and labels across a diverse collection of domains, including X-Ray, CT, MRI, dermoscopy, OCT, fundus photography, ultrasound, histopathology, and mammography. Rigorous evaluations demonstrate MedImageInsight's ability to achieve state-of-the-art (SOTA) or human expert level performance across classification, image-image search, and fine-tuning tasks. Specifically, on public datasets, MedImageInsight achieves SOTA in CT 3D medical image retrieval, as well as SOTA in disease classification and search for chest X-ray, dermatology, and OCT imaging. Furthermore, MedImageInsight achieves human expert performance in bone age estimation (on both public and partner data), as well as AUC above 0.9 in most other domains. When paired with a text decoder, MedImageInsight achieves near SOTA level single image report findings generation with less than 10\% the parameters of other models. Compared to fine-tuning GPT-4o with only MIMIC-CXR data for the same task, MedImageInsight outperforms in clinical metrics, but underperforms on lexical metrics where GPT-4o sets a new SOTA. Importantly for regulatory purposes, MedImageInsight can generate ROC curves, adjust sensitivity and specificity based on clinical need, and provide evidence-based decision support through image-image search (which can also enable retrieval augmented generation). In an independent clinical evaluation of image-image search in chest X-ray, MedImageInsight outperformed every other publicly available foundation model evaluated by large margins (over 6 points AUC), and significantly outperformed other models in terms of AI fairness (across age and gender). We hope releasing MedImageInsight will help enhance collective progress in medical imaging AI research and development. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_06542 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging Codella, Noel C. F. Jin, Ying Jain, Shrey Gu, Yu Lee, Ho Hin Abacha, Asma Ben Santamaria-Pang, Alberto Guyman, Will Sangani, Naiteek Zhang, Sheng Poon, Hoifung Hyland, Stephanie Bannur, Shruthi Alvarez-Valle, Javier Li, Xue Garrett, John McMillan, Alan Rajguru, Gaurav Maddi, Madhu Vijayrania, Nilesh Bhimai, Rehaan Mecklenburg, Nick Jain, Rupal Holstein, Daniel Gaur, Naveen Aski, Vijay Hwang, Jenq-Neng Lin, Thomas Tarapov, Ivan Lungren, Matthew Wei, Mu Image and Video Processing Computer Vision and Pattern Recognition In this work, we present MedImageInsight, an open-source medical imaging embedding model. MedImageInsight is trained on medical images with associated text and labels across a diverse collection of domains, including X-Ray, CT, MRI, dermoscopy, OCT, fundus photography, ultrasound, histopathology, and mammography. Rigorous evaluations demonstrate MedImageInsight's ability to achieve state-of-the-art (SOTA) or human expert level performance across classification, image-image search, and fine-tuning tasks. Specifically, on public datasets, MedImageInsight achieves SOTA in CT 3D medical image retrieval, as well as SOTA in disease classification and search for chest X-ray, dermatology, and OCT imaging. Furthermore, MedImageInsight achieves human expert performance in bone age estimation (on both public and partner data), as well as AUC above 0.9 in most other domains. When paired with a text decoder, MedImageInsight achieves near SOTA level single image report findings generation with less than 10\% the parameters of other models. Compared to fine-tuning GPT-4o with only MIMIC-CXR data for the same task, MedImageInsight outperforms in clinical metrics, but underperforms on lexical metrics where GPT-4o sets a new SOTA. Importantly for regulatory purposes, MedImageInsight can generate ROC curves, adjust sensitivity and specificity based on clinical need, and provide evidence-based decision support through image-image search (which can also enable retrieval augmented generation). In an independent clinical evaluation of image-image search in chest X-ray, MedImageInsight outperformed every other publicly available foundation model evaluated by large margins (over 6 points AUC), and significantly outperformed other models in terms of AI fairness (across age and gender). We hope releasing MedImageInsight will help enhance collective progress in medical imaging AI research and development. |
| title | MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging |
| topic | Image and Video Processing Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.06542 |