Multimodal Fusion on Low-quality Data: A Comprehensive Survey
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916463920545792 |
|---|---|
| author | Zhang, Qingyang Wei, Yake Han, Zongbo Fu, Huazhu Peng, Xi Deng, Cheng Hu, Qinghua Xu, Cai Wen, Jie Hu, Di Zhang, Changqing |
| author_facet | Zhang, Qingyang Wei, Yake Han, Zongbo Fu, Huazhu Peng, Xi Deng, Cheng Hu, Qinghua Xu, Cai Wen, Jie Hu, Di Zhang, Changqing |
| contents | Multimodal fusion focuses on integrating information from multiple modalities with the goal of more accurate prediction, which has achieved remarkable progress in a wide range of scenarios, including autonomous driving and medical diagnosis. However, the reliability of multimodal fusion remains largely unexplored especially under low-quality data settings. This paper surveys the common challenges and recent advances of multimodal fusion in the wild and presents them in a comprehensive taxonomy. From a data-centric view, we identify four main challenges that are faced by multimodal fusion on low-quality data, namely (1) noisy multimodal data that are contaminated with heterogeneous noises, (2) incomplete multimodal data that some modalities are missing, (3) imbalanced multimodal data that the qualities or properties of different modalities are significantly different and (4) quality-varying multimodal data that the quality of each modality dynamically changes with respect to different samples. This new taxonomy will enable researchers to understand the state of the field and identify several potential directions. We also provide discussion for the open problems in this field together with interesting future research directions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_18947 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multimodal Fusion on Low-quality Data: A Comprehensive Survey Zhang, Qingyang Wei, Yake Han, Zongbo Fu, Huazhu Peng, Xi Deng, Cheng Hu, Qinghua Xu, Cai Wen, Jie Hu, Di Zhang, Changqing Machine Learning Artificial Intelligence Multimodal fusion focuses on integrating information from multiple modalities with the goal of more accurate prediction, which has achieved remarkable progress in a wide range of scenarios, including autonomous driving and medical diagnosis. However, the reliability of multimodal fusion remains largely unexplored especially under low-quality data settings. This paper surveys the common challenges and recent advances of multimodal fusion in the wild and presents them in a comprehensive taxonomy. From a data-centric view, we identify four main challenges that are faced by multimodal fusion on low-quality data, namely (1) noisy multimodal data that are contaminated with heterogeneous noises, (2) incomplete multimodal data that some modalities are missing, (3) imbalanced multimodal data that the qualities or properties of different modalities are significantly different and (4) quality-varying multimodal data that the quality of each modality dynamically changes with respect to different samples. This new taxonomy will enable researchers to understand the state of the field and identify several potential directions. We also provide discussion for the open problems in this field together with interesting future research directions. |
| title | Multimodal Fusion on Low-quality Data: A Comprehensive Survey |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2404.18947 |