Generative Models with ELBOs Converging to Entropy Sums
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910786176155648 |
|---|---|
| author | Warnken, Jan Velychko, Dmytro Damm, Simon Fischer, Asja Lücke, Jörg |
| author_facet | Warnken, Jan Velychko, Dmytro Damm, Simon Fischer, Asja Lücke, Jörg |
| contents | The evidence lower bound (ELBO) is one of the most central objectives for probabilistic unsupervised learning. For the ELBOs of several generative models and model classes, we here prove convergence to entropy sums. As one result, we provide a list of generative models for which entropy convergence has been shown, so far, along with the corresponding expressions for entropy sums. Our considerations include very prominent generative models such as probabilistic PCA, sigmoid belief nets or Gaussian mixture models. However, we treat more models and entire model classes such as general mixtures of exponential family distributions. Our main contributions are the proofs for the individual models. For each given model we show that the conditions stated in Theorem 1 or Theorem 2 of [arXiv:2209.03077] are fulfilled such that by virtue of the theorems the given model's ELBO is equal to an entropy sum at all stationary points. The equality of the ELBO at stationary points applies under realistic conditions: for finite numbers of data points, for model/data mismatches, at any stationary point including saddle points etc, and it applies for any well behaved family of variational distributions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_09022 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Generative Models with ELBOs Converging to Entropy Sums Warnken, Jan Velychko, Dmytro Damm, Simon Fischer, Asja Lücke, Jörg Machine Learning Information Theory Probability Statistics Theory 65C20, 68T07, 60-08, 62-08, 62F99, 68T05 G.3 The evidence lower bound (ELBO) is one of the most central objectives for probabilistic unsupervised learning. For the ELBOs of several generative models and model classes, we here prove convergence to entropy sums. As one result, we provide a list of generative models for which entropy convergence has been shown, so far, along with the corresponding expressions for entropy sums. Our considerations include very prominent generative models such as probabilistic PCA, sigmoid belief nets or Gaussian mixture models. However, we treat more models and entire model classes such as general mixtures of exponential family distributions. Our main contributions are the proofs for the individual models. For each given model we show that the conditions stated in Theorem 1 or Theorem 2 of [arXiv:2209.03077] are fulfilled such that by virtue of the theorems the given model's ELBO is equal to an entropy sum at all stationary points. The equality of the ELBO at stationary points applies under realistic conditions: for finite numbers of data points, for model/data mismatches, at any stationary point including saddle points etc, and it applies for any well behaved family of variational distributions. |
| title | Generative Models with ELBOs Converging to Entropy Sums |
| topic | Machine Learning Information Theory Probability Statistics Theory 65C20, 68T07, 60-08, 62-08, 62F99, 68T05 G.3 |
| url | https://arxiv.org/abs/2501.09022 |