Interpretable Multimodal Cancer Prototyping with Whole Slide Images and Incompletely Paired Genomics
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918219286052864 |
|---|---|
| author | Zhang, Yupei Huang, Yating Hu, Wanming Yu, Lequan Yin, Hujun Li, Chao |
| author_facet | Zhang, Yupei Huang, Yating Hu, Wanming Yu, Lequan Yin, Hujun Li, Chao |
| contents | Multimodal approaches that integrate histology and genomics hold strong potential for precision oncology. However, phenotypic and genotypic heterogeneity limits the quality of intra-modal representations and hinders effective inter-modal integration. Furthermore, most existing methods overlook real-world clinical scenarios where genomics may be partially missing or entirely unavailable. We propose a flexible multimodal prototyping framework to integrate whole slide images and incomplete genomics for precision oncology. Our approach has four key components: 1) Biological Prototyping using text prompting and prototype-wise weighting; 2) Multiview Alignment through sample- and distribution-wise alignments; 3) Bipartite Fusion to capture both shared and modality-specific information for multimodal fusion; and 4) Semantic Genomics Imputation to handle missing data. Extensive experiments demonstrate the consistent superiority of the proposed method compared to other state-of-the-art approaches on multiple downstream tasks. The code is available at https://github.com/helenypzhang/Interpretable-Multimodal-Prototyping. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_21937 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Interpretable Multimodal Cancer Prototyping with Whole Slide Images and Incompletely Paired Genomics Zhang, Yupei Huang, Yating Hu, Wanming Yu, Lequan Yin, Hujun Li, Chao Computer Vision and Pattern Recognition Multimodal approaches that integrate histology and genomics hold strong potential for precision oncology. However, phenotypic and genotypic heterogeneity limits the quality of intra-modal representations and hinders effective inter-modal integration. Furthermore, most existing methods overlook real-world clinical scenarios where genomics may be partially missing or entirely unavailable. We propose a flexible multimodal prototyping framework to integrate whole slide images and incomplete genomics for precision oncology. Our approach has four key components: 1) Biological Prototyping using text prompting and prototype-wise weighting; 2) Multiview Alignment through sample- and distribution-wise alignments; 3) Bipartite Fusion to capture both shared and modality-specific information for multimodal fusion; and 4) Semantic Genomics Imputation to handle missing data. Extensive experiments demonstrate the consistent superiority of the proposed method compared to other state-of-the-art approaches on multiple downstream tasks. The code is available at https://github.com/helenypzhang/Interpretable-Multimodal-Prototyping. |
| title | Interpretable Multimodal Cancer Prototyping with Whole Slide Images and Incompletely Paired Genomics |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2511.21937 |