Enhancing Multimodal Unified Representations for Cross Modal Generalization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915316220559360 |
|---|---|
| author | Huang, Hai Xia, Yan Ji, Shengpeng Wang, Shulei Wang, Hanting Fang, Minghui Zhu, Jieming Dong, Zhenhua Zhou, Sashuai Zhao, Zhou |
| author_facet | Huang, Hai Xia, Yan Ji, Shengpeng Wang, Shulei Wang, Hanting Fang, Minghui Zhu, Jieming Dong, Zhenhua Zhou, Sashuai Zhao, Zhou |
| contents | To enhance the interpretability of multimodal unified representations, many studies have focused on discrete unified representations. These efforts typically start with contrastive learning and gradually extend to the disentanglement of modal information, achieving solid multimodal discrete unified representations. However, existing research often overlooks two critical issues: 1) The use of Euclidean distance for quantization in discrete representations often overlooks the important distinctions among different dimensions of features, resulting in redundant representations after quantization; 2) Different modalities have unique characteristics, and a uniform alignment approach does not fully exploit these traits. To address these issues, we propose Training-free Optimization of Codebook (TOC) and Fine and Coarse cross-modal Information Disentangling (FCID). These methods refine the unified discrete representations from pretraining and perform fine- and coarse-grained information disentanglement tailored to the specific characteristics of each modality, achieving significant performance improvements over previous state-of-the-art models. The code is available at https://github.com/haihuangcode/CMG. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_05168 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhancing Multimodal Unified Representations for Cross Modal Generalization Huang, Hai Xia, Yan Ji, Shengpeng Wang, Shulei Wang, Hanting Fang, Minghui Zhu, Jieming Dong, Zhenhua Zhou, Sashuai Zhao, Zhou Computer Vision and Pattern Recognition Artificial Intelligence To enhance the interpretability of multimodal unified representations, many studies have focused on discrete unified representations. These efforts typically start with contrastive learning and gradually extend to the disentanglement of modal information, achieving solid multimodal discrete unified representations. However, existing research often overlooks two critical issues: 1) The use of Euclidean distance for quantization in discrete representations often overlooks the important distinctions among different dimensions of features, resulting in redundant representations after quantization; 2) Different modalities have unique characteristics, and a uniform alignment approach does not fully exploit these traits. To address these issues, we propose Training-free Optimization of Codebook (TOC) and Fine and Coarse cross-modal Information Disentangling (FCID). These methods refine the unified discrete representations from pretraining and perform fine- and coarse-grained information disentanglement tailored to the specific characteristics of each modality, achieving significant performance improvements over previous state-of-the-art models. The code is available at https://github.com/haihuangcode/CMG. |
| title | Enhancing Multimodal Unified Representations for Cross Modal Generalization |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2403.05168 |