T3D: Advancing 3D Medical Vision-Language Pre-training by Learning Multi-View Visual Consistency
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917935391440896 |
|---|---|
| author | Liu, Che Ouyang, Cheng Chen, Yinda Quilodrán-Casas, Cesar César Ma, Lei Fu, Jie Guo, Yike Shah, Anand Bai, Wenjia Arcucci, Rossella |
| author_facet | Liu, Che Ouyang, Cheng Chen, Yinda Quilodrán-Casas, Cesar César Ma, Lei Fu, Jie Guo, Yike Shah, Anand Bai, Wenjia Arcucci, Rossella |
| contents | While 3D visual self-supervised learning (vSSL) shows promising results in capturing visual representations, it overlooks the clinical knowledge from radiology reports. Meanwhile, 3D medical vision-language pre-training (MedVLP) remains underexplored due to the lack of a large-scale, publicly available 3D medical image-report dataset. To bridge this gap, we introduce **CT-3DVLP**, the first and largest **public** 3D volume-report dataset, establishing a comprehensive benchmark for 3D MedVLP research. Meanwhile, we propose the **T3D** framework, which enhances 3D MedVLP beyond naive CLIP-style alignment that directly pairs volumes with reports but neglects local visual representations. Instead, we introduce **Text-informed Multi-view Alignment (TMA)**, a novel approach that clusters volumetric data while enforcing consistency across different views of the same volume-report pair. TMA integrates textual features into fine-grained visual representations, ensuring contextual coherence across views. We evaluate T3D across multiple downstream tasks in both unimodal and cross-modal settings, including zero-shot and fine-tuned classification, cross-modal retrieval, report generation, and semantic segmentation. Our results show that T3D consistently outperforms existing vSSL and multimodal methods, demonstrating superior zero-shot and fine-tuning capabilities and setting a new benchmark for 3D medical image understanding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_01529 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | T3D: Advancing 3D Medical Vision-Language Pre-training by Learning Multi-View Visual Consistency Liu, Che Ouyang, Cheng Chen, Yinda Quilodrán-Casas, Cesar César Ma, Lei Fu, Jie Guo, Yike Shah, Anand Bai, Wenjia Arcucci, Rossella Computer Vision and Pattern Recognition Computation and Language Machine Learning Image and Video Processing While 3D visual self-supervised learning (vSSL) shows promising results in capturing visual representations, it overlooks the clinical knowledge from radiology reports. Meanwhile, 3D medical vision-language pre-training (MedVLP) remains underexplored due to the lack of a large-scale, publicly available 3D medical image-report dataset. To bridge this gap, we introduce **CT-3DVLP**, the first and largest **public** 3D volume-report dataset, establishing a comprehensive benchmark for 3D MedVLP research. Meanwhile, we propose the **T3D** framework, which enhances 3D MedVLP beyond naive CLIP-style alignment that directly pairs volumes with reports but neglects local visual representations. Instead, we introduce **Text-informed Multi-view Alignment (TMA)**, a novel approach that clusters volumetric data while enforcing consistency across different views of the same volume-report pair. TMA integrates textual features into fine-grained visual representations, ensuring contextual coherence across views. We evaluate T3D across multiple downstream tasks in both unimodal and cross-modal settings, including zero-shot and fine-tuned classification, cross-modal retrieval, report generation, and semantic segmentation. Our results show that T3D consistently outperforms existing vSSL and multimodal methods, demonstrating superior zero-shot and fine-tuning capabilities and setting a new benchmark for 3D medical image understanding. |
| title | T3D: Advancing 3D Medical Vision-Language Pre-training by Learning Multi-View Visual Consistency |
| topic | Computer Vision and Pattern Recognition Computation and Language Machine Learning Image and Video Processing |
| url | https://arxiv.org/abs/2312.01529 |