Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.17834 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910014729355264 |
|---|---|
| author | Hamamci, Ibrahim Ethem Er, Sezgin Wang, Chenyu Almas, Furkan Simsek, Ayse Gulnihan Esirgun, Sevval Nil Dogan, Irem Durugol, Omer Faruk Hou, Benjamin Shit, Suprosanna Dai, Weicheng Xu, Murong Reynaud, Hadrien Dasdelen, Muhammed Furkan Wittmann, Bastian Amiranashvili, Tamaz Simsar, Enis Simsar, Mehmet Erdemir, Emine Bensu Alanbay, Abdullah Sekuboyina, Anjany Lafci, Berkan Kaplan, Ahmet Lu, Zhiyong Polacin, Malgorzata Kainz, Bernhard Bluethgen, Christian Batmanghelich, Kayhan Ozdemir, Mehmet Kemal Menze, Bjoern |
| author_facet | Hamamci, Ibrahim Ethem Er, Sezgin Wang, Chenyu Almas, Furkan Simsek, Ayse Gulnihan Esirgun, Sevval Nil Dogan, Irem Durugol, Omer Faruk Hou, Benjamin Shit, Suprosanna Dai, Weicheng Xu, Murong Reynaud, Hadrien Dasdelen, Muhammed Furkan Wittmann, Bastian Amiranashvili, Tamaz Simsar, Enis Simsar, Mehmet Erdemir, Emine Bensu Alanbay, Abdullah Sekuboyina, Anjany Lafci, Berkan Kaplan, Ahmet Lu, Zhiyong Polacin, Malgorzata Kainz, Bernhard Bluethgen, Christian Batmanghelich, Kayhan Ozdemir, Mehmet Kemal Menze, Bjoern |
| contents | Advancements in medical imaging AI, particularly in 3D imaging, have been limited due to the scarcity of comprehensive datasets. We introduce CT-RATE, a public dataset that pairs 3D medical images with corresponding textual reports. CT-RATE comprises 25,692 non-contrast 3D chest CT scans from 21,304 unique patients. Each scan is accompanied by its corresponding radiology report. Leveraging CT-RATE, we develop CT-CLIP, a CT-focused contrastive language-image pretraining framework designed for broad applications without the need for task-specific training. We demonstrate how CT-CLIP can be used in multi-abnormality detection and case retrieval, and outperforms state-of-the-art fully supervised models across all key metrics. By combining CT-CLIP's vision encoder with a pretrained large language model, we create CT-CHAT, a vision-language foundational chat model for 3D chest CT volumes. Finetuned on over 2.7 million question-answer pairs derived from the CT-RATE dataset, CT-CHAT underscores the necessity for specialized methods in 3D medical imaging. Collectively, the open-source release of CT-RATE, CT-CLIP, and CT-CHAT not only addresses critical challenges in 3D medical imaging but also lays the groundwork for future innovations in medical AI and improved patient care. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_17834 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography Hamamci, Ibrahim Ethem Er, Sezgin Wang, Chenyu Almas, Furkan Simsek, Ayse Gulnihan Esirgun, Sevval Nil Dogan, Irem Durugol, Omer Faruk Hou, Benjamin Shit, Suprosanna Dai, Weicheng Xu, Murong Reynaud, Hadrien Dasdelen, Muhammed Furkan Wittmann, Bastian Amiranashvili, Tamaz Simsar, Enis Simsar, Mehmet Erdemir, Emine Bensu Alanbay, Abdullah Sekuboyina, Anjany Lafci, Berkan Kaplan, Ahmet Lu, Zhiyong Polacin, Malgorzata Kainz, Bernhard Bluethgen, Christian Batmanghelich, Kayhan Ozdemir, Mehmet Kemal Menze, Bjoern Computer Vision and Pattern Recognition Advancements in medical imaging AI, particularly in 3D imaging, have been limited due to the scarcity of comprehensive datasets. We introduce CT-RATE, a public dataset that pairs 3D medical images with corresponding textual reports. CT-RATE comprises 25,692 non-contrast 3D chest CT scans from 21,304 unique patients. Each scan is accompanied by its corresponding radiology report. Leveraging CT-RATE, we develop CT-CLIP, a CT-focused contrastive language-image pretraining framework designed for broad applications without the need for task-specific training. We demonstrate how CT-CLIP can be used in multi-abnormality detection and case retrieval, and outperforms state-of-the-art fully supervised models across all key metrics. By combining CT-CLIP's vision encoder with a pretrained large language model, we create CT-CHAT, a vision-language foundational chat model for 3D chest CT volumes. Finetuned on over 2.7 million question-answer pairs derived from the CT-RATE dataset, CT-CHAT underscores the necessity for specialized methods in 3D medical imaging. Collectively, the open-source release of CT-RATE, CT-CLIP, and CT-CHAT not only addresses critical challenges in 3D medical imaging but also lays the groundwork for future innovations in medical AI and improved patient care. |
| title | Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.17834 |