Chameleon: Images Are What You Need For Multimodal Learning Robust To Missing Modalities
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911964884631552 |
|---|---|
| author | Liaqat, Muhammad Irzam Nawaz, Shah Zaheer, Muhammad Zaigham Saeed, Muhammad Saad Sajjad, Hassan De Schepper, Tom Nandakumar, Karthik Schedl, Muhammad Haris Khan Markus |
| author_facet | Liaqat, Muhammad Irzam Nawaz, Shah Zaheer, Muhammad Zaigham Saeed, Muhammad Saad Sajjad, Hassan De Schepper, Tom Nandakumar, Karthik Schedl, Muhammad Haris Khan Markus |
| contents | Multimodal learning has demonstrated remarkable performance improvements over unimodal architectures. However, multimodal learning methods often exhibit deteriorated performances if one or more modalities are missing. This may be attributed to the commonly used multi-branch design containing modality-specific streams making the models reliant on the availability of a complete set of modalities. In this work, we propose a robust textual-visual multimodal learning method, Chameleon, that completely deviates from the conventional multi-branch design. To enable this, we present the unification of input modalities into one format by encoding textual modality into visual representations. As a result, our approach does not require modality-specific branches to learn modality-independent multimodal representations making it robust to missing modalities. Extensive experiments are performed on four popular challenging datasets including Hateful Memes, UPMC Food-101, MM-IMDb, and Ferramenta. Chameleon not only achieves superior performance when all modalities are present at train/test time but also demonstrates notable resilience in the case of missing modalities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_16243 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Chameleon: Images Are What You Need For Multimodal Learning Robust To Missing Modalities Liaqat, Muhammad Irzam Nawaz, Shah Zaheer, Muhammad Zaigham Saeed, Muhammad Saad Sajjad, Hassan De Schepper, Tom Nandakumar, Karthik Schedl, Muhammad Haris Khan Markus Computer Vision and Pattern Recognition Multimodal learning has demonstrated remarkable performance improvements over unimodal architectures. However, multimodal learning methods often exhibit deteriorated performances if one or more modalities are missing. This may be attributed to the commonly used multi-branch design containing modality-specific streams making the models reliant on the availability of a complete set of modalities. In this work, we propose a robust textual-visual multimodal learning method, Chameleon, that completely deviates from the conventional multi-branch design. To enable this, we present the unification of input modalities into one format by encoding textual modality into visual representations. As a result, our approach does not require modality-specific branches to learn modality-independent multimodal representations making it robust to missing modalities. Extensive experiments are performed on four popular challenging datasets including Hateful Memes, UPMC Food-101, MM-IMDb, and Ferramenta. Chameleon not only achieves superior performance when all modalities are present at train/test time but also demonstrates notable resilience in the case of missing modalities. |
| title | Chameleon: Images Are What You Need For Multimodal Learning Robust To Missing Modalities |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.16243 |