Efficient Distribution Matching of Representations via Noise-Injected Deep InfoMax
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909456290283520 |
|---|---|
| author | Butakov, Ivan Semenenko, Alexander Tolmachev, Alexander Gladkov, Andrey Munkhoeva, Marina Frolov, Alexey |
| author_facet | Butakov, Ivan Semenenko, Alexander Tolmachev, Alexander Gladkov, Andrey Munkhoeva, Marina Frolov, Alexey |
| contents | Deep InfoMax (DIM) is a well-established method for self-supervised representation learning (SSRL) based on maximization of the mutual information between the input and the output of a deep neural network encoder. Despite the DIM and contrastive SSRL in general being well-explored, the task of learning representations conforming to a specific distribution (i.e., distribution matching, DM) is still under-addressed. Motivated by the importance of DM to several downstream tasks (including generative modeling, disentanglement, outliers detection and other), we enhance DIM to enable automatic matching of learned representations to a selected prior distribution. To achieve this, we propose injecting an independent noise into the normalized outputs of the encoder, while keeping the same InfoMax training objective. We show that such modification allows for learning uniformly and normally distributed representations, as well as representations of other absolutely continuous distributions. Our approach is tested on various downstream tasks. The results indicate a moderate trade-off between the performance on the downstream tasks and quality of DM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_06993 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Efficient Distribution Matching of Representations via Noise-Injected Deep InfoMax Butakov, Ivan Semenenko, Alexander Tolmachev, Alexander Gladkov, Andrey Munkhoeva, Marina Frolov, Alexey Machine Learning Information Theory 94A16 (Primary) 68T07, 94A17 (Secondary) E.4; H.1.1 Deep InfoMax (DIM) is a well-established method for self-supervised representation learning (SSRL) based on maximization of the mutual information between the input and the output of a deep neural network encoder. Despite the DIM and contrastive SSRL in general being well-explored, the task of learning representations conforming to a specific distribution (i.e., distribution matching, DM) is still under-addressed. Motivated by the importance of DM to several downstream tasks (including generative modeling, disentanglement, outliers detection and other), we enhance DIM to enable automatic matching of learned representations to a selected prior distribution. To achieve this, we propose injecting an independent noise into the normalized outputs of the encoder, while keeping the same InfoMax training objective. We show that such modification allows for learning uniformly and normally distributed representations, as well as representations of other absolutely continuous distributions. Our approach is tested on various downstream tasks. The results indicate a moderate trade-off between the performance on the downstream tasks and quality of DM. |
| title | Efficient Distribution Matching of Representations via Noise-Injected Deep InfoMax |
| topic | Machine Learning Information Theory 94A16 (Primary) 68T07, 94A17 (Secondary) E.4; H.1.1 |
| url | https://arxiv.org/abs/2410.06993 |