Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.13300 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913550685962240 |
|---|---|
| author | Soletskyi, Roman Gabrié, Marylou Loureiro, Bruno |
| author_facet | Soletskyi, Roman Gabrié, Marylou Loureiro, Bruno |
| contents | While deep learning has expanded the possibilities for highly expressive variational families, the practical benefits of these tools for variational inference (VI) are often limited by the minimization of the traditional Kullback-Leibler objective, which can yield suboptimal solutions. A major challenge in this context is \emph{mode collapse}: the phenomenon where a model concentrates on a few modes of the target distribution during training, despite being statistically capable of expressing them all. In this work, we carry a theoretical investigation of mode collapse for the gradient flow on Gaussian mixture models. We identify the key low-dimensional statistics characterizing the flow, and derive a closed set of low-dimensional equations governing their evolution. Leveraging this compact description, we show that mode collapse is present even in statistically favorable scenarios, and identify two key mechanisms driving it: mean alignment and vanishing weight. Our theoretical findings are consistent with the implementation of VI using normalizing flows, a class of popular generative models, thereby offering practical insights. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_13300 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A theoretical perspective on mode collapse in variational inference Soletskyi, Roman Gabrié, Marylou Loureiro, Bruno Machine Learning Disordered Systems and Neural Networks Statistical Mechanics Statistics Theory While deep learning has expanded the possibilities for highly expressive variational families, the practical benefits of these tools for variational inference (VI) are often limited by the minimization of the traditional Kullback-Leibler objective, which can yield suboptimal solutions. A major challenge in this context is \emph{mode collapse}: the phenomenon where a model concentrates on a few modes of the target distribution during training, despite being statistically capable of expressing them all. In this work, we carry a theoretical investigation of mode collapse for the gradient flow on Gaussian mixture models. We identify the key low-dimensional statistics characterizing the flow, and derive a closed set of low-dimensional equations governing their evolution. Leveraging this compact description, we show that mode collapse is present even in statistically favorable scenarios, and identify two key mechanisms driving it: mean alignment and vanishing weight. Our theoretical findings are consistent with the implementation of VI using normalizing flows, a class of popular generative models, thereby offering practical insights. |
| title | A theoretical perspective on mode collapse in variational inference |
| topic | Machine Learning Disordered Systems and Neural Networks Statistical Mechanics Statistics Theory |
| url | https://arxiv.org/abs/2410.13300 |