Knowledge Bridger: Towards Training-free Missing Modality Completion
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909651159744512 |
|---|---|
| author | Ke, Guanzhou He, Shengfeng Wang, Xiao Li Wang, Bo Chao, Guoqing Zhang, Yuanyang Xie, Yi Su, HeXing |
| author_facet | Ke, Guanzhou He, Shengfeng Wang, Xiao Li Wang, Bo Chao, Guoqing Zhang, Yuanyang Xie, Yi Su, HeXing |
| contents | Previous successful approaches to missing modality completion rely on carefully designed fusion techniques and extensive pre-training on complete data, which can limit their generalizability in out-of-domain (OOD) scenarios. In this study, we pose a new challenge: can we develop a missing modality completion model that is both resource-efficient and robust to OOD generalization? To address this, we present a training-free framework for missing modality completion that leverages large multimodal models (LMMs). Our approach, termed the "Knowledge Bridger", is modality-agnostic and integrates generation and ranking of missing modalities. By defining domain-specific priors, our method automatically extracts structured information from available modalities to construct knowledge graphs. These extracted graphs connect the missing modality generation and ranking modules through the LMM, resulting in high-quality imputations of missing modalities. Experimental results across both general and medical domains show that our approach consistently outperforms competing methods, including in OOD generalization. Additionally, our knowledge-driven generation and ranking techniques demonstrate superiority over variants that directly employ LMMs for generation and ranking, offering insights that may be valuable for applications in other domains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_19834 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Knowledge Bridger: Towards Training-free Missing Modality Completion Ke, Guanzhou He, Shengfeng Wang, Xiao Li Wang, Bo Chao, Guoqing Zhang, Yuanyang Xie, Yi Su, HeXing Machine Learning Computer Vision and Pattern Recognition Multimedia Previous successful approaches to missing modality completion rely on carefully designed fusion techniques and extensive pre-training on complete data, which can limit their generalizability in out-of-domain (OOD) scenarios. In this study, we pose a new challenge: can we develop a missing modality completion model that is both resource-efficient and robust to OOD generalization? To address this, we present a training-free framework for missing modality completion that leverages large multimodal models (LMMs). Our approach, termed the "Knowledge Bridger", is modality-agnostic and integrates generation and ranking of missing modalities. By defining domain-specific priors, our method automatically extracts structured information from available modalities to construct knowledge graphs. These extracted graphs connect the missing modality generation and ranking modules through the LMM, resulting in high-quality imputations of missing modalities. Experimental results across both general and medical domains show that our approach consistently outperforms competing methods, including in OOD generalization. Additionally, our knowledge-driven generation and ranking techniques demonstrate superiority over variants that directly employ LMMs for generation and ranking, offering insights that may be valuable for applications in other domains. |
| title | Knowledge Bridger: Towards Training-free Missing Modality Completion |
| topic | Machine Learning Computer Vision and Pattern Recognition Multimedia |
| url | https://arxiv.org/abs/2502.19834 |