Comprehensive language-image pre-training for 3D medical image understanding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915729146642432 |
|---|---|
| author | Wald, Tassilo Hamamci, Ibrahim Ethem Gao, Yuan Bond-Taylor, Sam Sharma, Harshita Ilse, Maximilian Lo, Cynthia Melnichenko, Olesya Schwaighofer, Anton Codella, Noel C. F. Wetscherek, Maria Teodora Maier-Hein, Klaus H. Korfiatis, Panagiotis Salvatelli, Valentina Alvarez-Valle, Javier Pérez-García, Fernando |
| author_facet | Wald, Tassilo Hamamci, Ibrahim Ethem Gao, Yuan Bond-Taylor, Sam Sharma, Harshita Ilse, Maximilian Lo, Cynthia Melnichenko, Olesya Schwaighofer, Anton Codella, Noel C. F. Wetscherek, Maria Teodora Maier-Hein, Klaus H. Korfiatis, Panagiotis Salvatelli, Valentina Alvarez-Valle, Javier Pérez-García, Fernando |
| contents | Vision-language pre-training, i.e., aligning images with paired text, is a powerful paradigm to create encoders that can be directly used for tasks such as classification, retrieval, and segmentation. In the 3D medical image domain, these capabilities allow vision-language encoders (VLEs) to support radiologists by retrieving patients with similar abnormalities, predicting likelihoods of abnormality, or, with downstream adaptation, generating radiological reports. While the methodology holds promise, data availability and domain-specific hurdles limit the capabilities of current 3D VLEs.
In this paper, we overcome these challenges by injecting additional supervision via a report generation objective and combining vision-language with vision-only pre-training. This allows us to leverage both image-only and paired image-text 3D datasets, increasing the total amount of data to which our model is exposed. Through these additional objectives, paired with best practices of the 3D medical imaging domain, we develop the Comprehensive Language-Image Pre-training (COLIPRI) encoder family. Our COLIPRI encoders achieve state-of-the-art performance in report generation, semantic segmentation, classification probing, and zero-shot classification. The model is available at https://huggingface.co/microsoft/colipri. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_15042 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Comprehensive language-image pre-training for 3D medical image understanding Wald, Tassilo Hamamci, Ibrahim Ethem Gao, Yuan Bond-Taylor, Sam Sharma, Harshita Ilse, Maximilian Lo, Cynthia Melnichenko, Olesya Schwaighofer, Anton Codella, Noel C. F. Wetscherek, Maria Teodora Maier-Hein, Klaus H. Korfiatis, Panagiotis Salvatelli, Valentina Alvarez-Valle, Javier Pérez-García, Fernando Computer Vision and Pattern Recognition Machine Learning Vision-language pre-training, i.e., aligning images with paired text, is a powerful paradigm to create encoders that can be directly used for tasks such as classification, retrieval, and segmentation. In the 3D medical image domain, these capabilities allow vision-language encoders (VLEs) to support radiologists by retrieving patients with similar abnormalities, predicting likelihoods of abnormality, or, with downstream adaptation, generating radiological reports. While the methodology holds promise, data availability and domain-specific hurdles limit the capabilities of current 3D VLEs. In this paper, we overcome these challenges by injecting additional supervision via a report generation objective and combining vision-language with vision-only pre-training. This allows us to leverage both image-only and paired image-text 3D datasets, increasing the total amount of data to which our model is exposed. Through these additional objectives, paired with best practices of the 3D medical imaging domain, we develop the Comprehensive Language-Image Pre-training (COLIPRI) encoder family. Our COLIPRI encoders achieve state-of-the-art performance in report generation, semantic segmentation, classification probing, and zero-shot classification. The model is available at https://huggingface.co/microsoft/colipri. |
| title | Comprehensive language-image pre-training for 3D medical image understanding |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2510.15042 |