Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.09151 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913107414089728 |
|---|---|
| author | Li, Frank Khosravi, Bardia Chavoshi, Mohammadreza Jeon, Young Seok Dapamede, Theo Trivedi, Hari Newsome, Janice Gichoya, Judy |
| author_facet | Li, Frank Khosravi, Bardia Chavoshi, Mohammadreza Jeon, Young Seok Dapamede, Theo Trivedi, Hari Newsome, Janice Gichoya, Judy |
| contents | Multi-modal medical imaging enables comprehensive diagnostics, yet current foundation models process 2D (e.g. X-ray) and 3D (e.g. CT) data with separate, dimensionality-specific architectures. We present MultiMedVision, a unified framework for joint 2D/3D representation learning built on a Sparse Vision Transformer. Our model uses 3D Rotary Positional Embeddings and variable-length sequence packing to process mixed-modality batches natively within a shared latent space, without modality-specific adapters or treating 3D volumes as 2D slice sequences. Trained with a self-supervised objective on chest X-rays (MIMIC-CXR) and CT scans (CT-RATE), and using a single shared encoder with 5x less data, MultiMedVision achieves competitive performance on both 2D benchmarks (Macro AUROC 0.82 on MIMIC, 0.84 on CheXpert) and 3D tasks (0.85 on CT-RATE). Analysis of the learned representations reveals coexisting modality-specific and shared feature subspaces, demonstrating that unified cross-dimensional representation learning is feasible without sacrificing modality-specific performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_09151 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MultiMedVision: Multi-Modal Medical Vision Framework Li, Frank Khosravi, Bardia Chavoshi, Mohammadreza Jeon, Young Seok Dapamede, Theo Trivedi, Hari Newsome, Janice Gichoya, Judy Computer Vision and Pattern Recognition Multi-modal medical imaging enables comprehensive diagnostics, yet current foundation models process 2D (e.g. X-ray) and 3D (e.g. CT) data with separate, dimensionality-specific architectures. We present MultiMedVision, a unified framework for joint 2D/3D representation learning built on a Sparse Vision Transformer. Our model uses 3D Rotary Positional Embeddings and variable-length sequence packing to process mixed-modality batches natively within a shared latent space, without modality-specific adapters or treating 3D volumes as 2D slice sequences. Trained with a self-supervised objective on chest X-rays (MIMIC-CXR) and CT scans (CT-RATE), and using a single shared encoder with 5x less data, MultiMedVision achieves competitive performance on both 2D benchmarks (Macro AUROC 0.82 on MIMIC, 0.84 on CheXpert) and 3D tasks (0.85 on CT-RATE). Analysis of the learned representations reveals coexisting modality-specific and shared feature subspaces, demonstrating that unified cross-dimensional representation learning is feasible without sacrificing modality-specific performance. |
| title | MultiMedVision: Multi-Modal Medical Vision Framework |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2605.09151 |