Disentangled and Interpretable Multimodal Attention Fusion for Cancer Survival Prediction
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915360841662464 |
|---|---|
| author | Eijpe, Aniek Lakbir, Soufyan Cesur, Melis Erdal Oliveira, Sara P. Abeln, Sanne Silva, Wilson |
| author_facet | Eijpe, Aniek Lakbir, Soufyan Cesur, Melis Erdal Oliveira, Sara P. Abeln, Sanne Silva, Wilson |
| contents | To improve the prediction of cancer survival using whole-slide images and transcriptomics data, it is crucial to capture both modality-shared and modality-specific information. However, multimodal frameworks often entangle these representations, limiting interpretability and potentially suppressing discriminative features. To address this, we propose Disentangled and Interpretable Multimodal Attention Fusion (DIMAF), a multimodal framework that separates the intra- and inter-modal interactions within an attention-based fusion mechanism to learn distinct modality-specific and modality-shared representations. We introduce a loss based on Distance Correlation to promote disentanglement between these representations and integrate Shapley additive explanations to assess their relative contributions to survival prediction. We evaluate DIMAF on four public cancer survival datasets, achieving a relative average improvement of 1.85% in performance and 23.7% in disentanglement compared to current state-of-the-art multimodal models. Beyond improved performance, our interpretable framework enables a deeper exploration of the underlying interactions between and within modalities in cancer biology. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_16069 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Disentangled and Interpretable Multimodal Attention Fusion for Cancer Survival Prediction Eijpe, Aniek Lakbir, Soufyan Cesur, Melis Erdal Oliveira, Sara P. Abeln, Sanne Silva, Wilson Computer Vision and Pattern Recognition To improve the prediction of cancer survival using whole-slide images and transcriptomics data, it is crucial to capture both modality-shared and modality-specific information. However, multimodal frameworks often entangle these representations, limiting interpretability and potentially suppressing discriminative features. To address this, we propose Disentangled and Interpretable Multimodal Attention Fusion (DIMAF), a multimodal framework that separates the intra- and inter-modal interactions within an attention-based fusion mechanism to learn distinct modality-specific and modality-shared representations. We introduce a loss based on Distance Correlation to promote disentanglement between these representations and integrate Shapley additive explanations to assess their relative contributions to survival prediction. We evaluate DIMAF on four public cancer survival datasets, achieving a relative average improvement of 1.85% in performance and 23.7% in disentanglement compared to current state-of-the-art multimodal models. Beyond improved performance, our interpretable framework enables a deeper exploration of the underlying interactions between and within modalities in cancer biology. |
| title | Disentangled and Interpretable Multimodal Attention Fusion for Cancer Survival Prediction |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2503.16069 |