Merlin: A Computed Tomography Vision-Language Foundation Model and Dataset
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915832471224320 |
|---|---|
| author | Blankemeier, Louis Kumar, Ashwin Cohen, Joseph Paul Liu, Jiaming Liu, Longchao Van Veen, Dave Gardezi, Syed Jamal Safdar Yu, Hongkun Paschali, Magdalini Chen, Zhihong Delbrouck, Jean-Benoit Reis, Eduardo Holland, Robbie Truyts, Cesar Bluethgen, Christian Wu, Yufu Lian, Long Jensen, Malte Engmann Kjeldskov Ostmeier, Sophie Varma, Maya Valanarasu, Jeya Maria Jose Fang, Zhongnan Huo, Zepeng Nabulsi, Zaid Ardila, Diego Weng, Wei-Hung Junior, Edson Amaro Ahuja, Neera Fries, Jason Shah, Nigam H. Zaharchuk, Greg Willis, Marc Yala, Adam Johnston, Andrew Boutin, Robert D. Wentland, Andrew Langlotz, Curtis P. Hom, Jason Gatidis, Sergios Chaudhari, Akshay S. |
| author_facet | Blankemeier, Louis Kumar, Ashwin Cohen, Joseph Paul Liu, Jiaming Liu, Longchao Van Veen, Dave Gardezi, Syed Jamal Safdar Yu, Hongkun Paschali, Magdalini Chen, Zhihong Delbrouck, Jean-Benoit Reis, Eduardo Holland, Robbie Truyts, Cesar Bluethgen, Christian Wu, Yufu Lian, Long Jensen, Malte Engmann Kjeldskov Ostmeier, Sophie Varma, Maya Valanarasu, Jeya Maria Jose Fang, Zhongnan Huo, Zepeng Nabulsi, Zaid Ardila, Diego Weng, Wei-Hung Junior, Edson Amaro Ahuja, Neera Fries, Jason Shah, Nigam H. Zaharchuk, Greg Willis, Marc Yala, Adam Johnston, Andrew Boutin, Robert D. Wentland, Andrew Langlotz, Curtis P. Hom, Jason Gatidis, Sergios Chaudhari, Akshay S. |
| contents | The large volume of abdominal computed tomography (CT) scans coupled with the shortage of radiologists have intensified the need for automated medical image analysis tools. Previous state-of-the-art approaches for automated analysis leverage vision-language models (VLMs) that jointly model images and radiology reports. However, current medical VLMs are generally limited to 2D images and short reports. Here to overcome these shortcomings for abdominal CT interpretation, we introduce Merlin, a 3D VLM that learns from volumetric CT scans, electronic health record data and radiology reports. This approach is enabled by a multistage pretraining framework that does not require additional manual annotations. We trained Merlin using a high-quality clinical dataset of paired CT scans (>6 million images from 15,331 CT scans), diagnosis codes (>1.8 million codes) and radiology reports (>6 million tokens). We comprehensively evaluated Merlin on 6 task types and 752 individual tasks that covered diagnostic, prognostic and quality-related tasks. The non-adapted (off-the-shelf) tasks included zero-shot classification of findings (30 findings), phenotype classification (692 phenotypes) and zero-shot cross-modal retrieval (image-to-findings and image-to-impression). The model-adapted tasks included 5-year chronic disease prediction (6 diseases), radiology report generation and 3D semantic segmentation (20 organs). We validated Merlin at scale, with internal testing on 5,137 CT scans and external testing on 44,098 CT scans from 3 independent sites and 2 public datasets. The results demonstrated high generalization across institutions and anatomies. Merlin outperformed 2D VLMs, CT foundation models and off-the-shelf radiology models. We also release our trained models, code, and dataset, available at: https://github.com/StanfordMIMI/Merlin. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_06512 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Merlin: A Computed Tomography Vision-Language Foundation Model and Dataset Blankemeier, Louis Kumar, Ashwin Cohen, Joseph Paul Liu, Jiaming Liu, Longchao Van Veen, Dave Gardezi, Syed Jamal Safdar Yu, Hongkun Paschali, Magdalini Chen, Zhihong Delbrouck, Jean-Benoit Reis, Eduardo Holland, Robbie Truyts, Cesar Bluethgen, Christian Wu, Yufu Lian, Long Jensen, Malte Engmann Kjeldskov Ostmeier, Sophie Varma, Maya Valanarasu, Jeya Maria Jose Fang, Zhongnan Huo, Zepeng Nabulsi, Zaid Ardila, Diego Weng, Wei-Hung Junior, Edson Amaro Ahuja, Neera Fries, Jason Shah, Nigam H. Zaharchuk, Greg Willis, Marc Yala, Adam Johnston, Andrew Boutin, Robert D. Wentland, Andrew Langlotz, Curtis P. Hom, Jason Gatidis, Sergios Chaudhari, Akshay S. Computer Vision and Pattern Recognition Artificial Intelligence The large volume of abdominal computed tomography (CT) scans coupled with the shortage of radiologists have intensified the need for automated medical image analysis tools. Previous state-of-the-art approaches for automated analysis leverage vision-language models (VLMs) that jointly model images and radiology reports. However, current medical VLMs are generally limited to 2D images and short reports. Here to overcome these shortcomings for abdominal CT interpretation, we introduce Merlin, a 3D VLM that learns from volumetric CT scans, electronic health record data and radiology reports. This approach is enabled by a multistage pretraining framework that does not require additional manual annotations. We trained Merlin using a high-quality clinical dataset of paired CT scans (>6 million images from 15,331 CT scans), diagnosis codes (>1.8 million codes) and radiology reports (>6 million tokens). We comprehensively evaluated Merlin on 6 task types and 752 individual tasks that covered diagnostic, prognostic and quality-related tasks. The non-adapted (off-the-shelf) tasks included zero-shot classification of findings (30 findings), phenotype classification (692 phenotypes) and zero-shot cross-modal retrieval (image-to-findings and image-to-impression). The model-adapted tasks included 5-year chronic disease prediction (6 diseases), radiology report generation and 3D semantic segmentation (20 organs). We validated Merlin at scale, with internal testing on 5,137 CT scans and external testing on 44,098 CT scans from 3 independent sites and 2 public datasets. The results demonstrated high generalization across institutions and anatomies. Merlin outperformed 2D VLMs, CT foundation models and off-the-shelf radiology models. We also release our trained models, code, and dataset, available at: https://github.com/StanfordMIMI/Merlin. |
| title | Merlin: A Computed Tomography Vision-Language Foundation Model and Dataset |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2406.06512 |