An Information Criterion for Controlled Disentanglement of Multimodal Data
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915201108934656 |
|---|---|
| author | Wang, Chenyu Gupta, Sharut Zhang, Xinyi Tonekaboni, Sana Jegelka, Stefanie Jaakkola, Tommi Uhler, Caroline |
| author_facet | Wang, Chenyu Gupta, Sharut Zhang, Xinyi Tonekaboni, Sana Jegelka, Stefanie Jaakkola, Tommi Uhler, Caroline |
| contents | Multimodal representation learning seeks to relate and decompose information inherent in multiple modalities. By disentangling modality-specific information from information that is shared across modalities, we can improve interpretability and robustness and enable downstream tasks such as the generation of counterfactual outcomes. Separating the two types of information is challenging since they are often deeply entangled in many real-world applications. We propose Disentangled Self-Supervised Learning (DisentangledSSL), a novel self-supervised approach for learning disentangled representations. We present a comprehensive analysis of the optimality of each disentangled representation, particularly focusing on the scenario not covered in prior work where the so-called Minimum Necessary Information (MNI) point is not attainable. We demonstrate that DisentangledSSL successfully learns shared and modality-specific features on multiple synthetic and real-world datasets and consistently outperforms baselines on various downstream tasks, including prediction tasks for vision-language data, as well as molecule-phenotype retrieval tasks for biological data. The code is available at https://github.com/uhlerlab/DisentangledSSL. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_23996 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | An Information Criterion for Controlled Disentanglement of Multimodal Data Wang, Chenyu Gupta, Sharut Zhang, Xinyi Tonekaboni, Sana Jegelka, Stefanie Jaakkola, Tommi Uhler, Caroline Machine Learning Artificial Intelligence Information Theory Multimodal representation learning seeks to relate and decompose information inherent in multiple modalities. By disentangling modality-specific information from information that is shared across modalities, we can improve interpretability and robustness and enable downstream tasks such as the generation of counterfactual outcomes. Separating the two types of information is challenging since they are often deeply entangled in many real-world applications. We propose Disentangled Self-Supervised Learning (DisentangledSSL), a novel self-supervised approach for learning disentangled representations. We present a comprehensive analysis of the optimality of each disentangled representation, particularly focusing on the scenario not covered in prior work where the so-called Minimum Necessary Information (MNI) point is not attainable. We demonstrate that DisentangledSSL successfully learns shared and modality-specific features on multiple synthetic and real-world datasets and consistently outperforms baselines on various downstream tasks, including prediction tasks for vision-language data, as well as molecule-phenotype retrieval tasks for biological data. The code is available at https://github.com/uhlerlab/DisentangledSSL. |
| title | An Information Criterion for Controlled Disentanglement of Multimodal Data |
| topic | Machine Learning Artificial Intelligence Information Theory |
| url | https://arxiv.org/abs/2410.23996 |