Aligning Multimodal Representations through an Information Bottleneck
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908394669998080 |
|---|---|
| author | Almudévar, Antonio Hernández-Lobato, José Miguel Khurana, Sameer Marxer, Ricard Ortega, Alfonso |
| author_facet | Almudévar, Antonio Hernández-Lobato, José Miguel Khurana, Sameer Marxer, Ricard Ortega, Alfonso |
| contents | Contrastive losses have been extensively used as a tool for multimodal representation learning. However, it has been empirically observed that their use is not effective to learn an aligned representation space. In this paper, we argue that this phenomenon is caused by the presence of modality-specific information in the representation space. Although some of the most widely used contrastive losses maximize the mutual information between representations of both modalities, they are not designed to remove the modality-specific information. We give a theoretical description of this problem through the lens of the Information Bottleneck Principle. We also empirically analyze how different hyperparameters affect the emergence of this phenomenon in a controlled experimental setup. Finally, we propose a regularization term in the loss function that is derived by means of a variational approximation and aims to increase the representational alignment. We analyze in a set of controlled experiments and real-world applications the advantages of including this regularization term. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_04870 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Aligning Multimodal Representations through an Information Bottleneck Almudévar, Antonio Hernández-Lobato, José Miguel Khurana, Sameer Marxer, Ricard Ortega, Alfonso Machine Learning Contrastive losses have been extensively used as a tool for multimodal representation learning. However, it has been empirically observed that their use is not effective to learn an aligned representation space. In this paper, we argue that this phenomenon is caused by the presence of modality-specific information in the representation space. Although some of the most widely used contrastive losses maximize the mutual information between representations of both modalities, they are not designed to remove the modality-specific information. We give a theoretical description of this problem through the lens of the Information Bottleneck Principle. We also empirically analyze how different hyperparameters affect the emergence of this phenomenon in a controlled experimental setup. Finally, we propose a regularization term in the loss function that is derived by means of a variational approximation and aims to increase the representational alignment. We analyze in a set of controlled experiments and real-world applications the advantages of including this regularization term. |
| title | Aligning Multimodal Representations through an Information Bottleneck |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2506.04870 |