Unity by Diversity: Improved Representation Learning in Multimodal VAEs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909450165551104 |
|---|---|
| author | Sutter, Thomas M. Meng, Yang Agostini, Andrea Chopard, Daphné Fortin, Norbert Vogt, Julia E. Shahbaba, Babak Mandt, Stephan |
| author_facet | Sutter, Thomas M. Meng, Yang Agostini, Andrea Chopard, Daphné Fortin, Norbert Vogt, Julia E. Shahbaba, Babak Mandt, Stephan |
| contents | Variational Autoencoders for multimodal data hold promise for many tasks in data analysis, such as representation learning, conditional generation, and imputation. Current architectures either share the encoder output, decoder input, or both across modalities to learn a shared representation. Such architectures impose hard constraints on the model. In this work, we show that a better latent representation can be obtained by replacing these hard constraints with a soft constraint. We propose a new mixture-of-experts prior, softly guiding each modality's latent representation towards a shared aggregate posterior. This approach results in a superior latent representation and allows each encoding to preserve information better from its uncompressed original features. In extensive experiments on multiple benchmark datasets and two challenging real-world datasets, we show improved learned latent representations and imputation of missing data modalities compared to existing methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_05300 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Unity by Diversity: Improved Representation Learning in Multimodal VAEs Sutter, Thomas M. Meng, Yang Agostini, Andrea Chopard, Daphné Fortin, Norbert Vogt, Julia E. Shahbaba, Babak Mandt, Stephan Machine Learning Artificial Intelligence Variational Autoencoders for multimodal data hold promise for many tasks in data analysis, such as representation learning, conditional generation, and imputation. Current architectures either share the encoder output, decoder input, or both across modalities to learn a shared representation. Such architectures impose hard constraints on the model. In this work, we show that a better latent representation can be obtained by replacing these hard constraints with a soft constraint. We propose a new mixture-of-experts prior, softly guiding each modality's latent representation towards a shared aggregate posterior. This approach results in a superior latent representation and allows each encoding to preserve information better from its uncompressed original features. In extensive experiments on multiple benchmark datasets and two challenging real-world datasets, we show improved learned latent representations and imputation of missing data modalities compared to existing methods. |
| title | Unity by Diversity: Improved Representation Learning in Multimodal VAEs |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2403.05300 |