Reparameterization invariance in approximate Bayesian inference
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929708409552896 |
|---|---|
| author | Roy, Hrittik Miani, Marco Ek, Carl Henrik Hennig, Philipp Pförtner, Marvin Tatzel, Lukas Hauberg, Søren |
| author_facet | Roy, Hrittik Miani, Marco Ek, Carl Henrik Hennig, Philipp Pförtner, Marvin Tatzel, Lukas Hauberg, Søren |
| contents | Current approximate posteriors in Bayesian neural networks (BNNs) exhibit a crucial limitation: they fail to maintain invariance under reparameterization, i.e. BNNs assign different posterior densities to different parametrizations of identical functions. This creates a fundamental flaw in the application of Bayesian principles as it breaks the correspondence between uncertainty over the parameters with uncertainty over the parametrized function. In this paper, we investigate this issue in the context of the increasingly popular linearized Laplace approximation. Specifically, it has been observed that linearized predictives alleviate the common underfitting problems of the Laplace approximation. We develop a new geometric view of reparametrizations from which we explain the success of linearization. Moreover, we demonstrate that these reparameterization invariance properties can be extended to the original neural network predictive using a Riemannian diffusion process giving a straightforward algorithm for approximate posterior sampling, which empirically improves posterior fit. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_03334 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Reparameterization invariance in approximate Bayesian inference Roy, Hrittik Miani, Marco Ek, Carl Henrik Hennig, Philipp Pförtner, Marvin Tatzel, Lukas Hauberg, Søren Machine Learning Current approximate posteriors in Bayesian neural networks (BNNs) exhibit a crucial limitation: they fail to maintain invariance under reparameterization, i.e. BNNs assign different posterior densities to different parametrizations of identical functions. This creates a fundamental flaw in the application of Bayesian principles as it breaks the correspondence between uncertainty over the parameters with uncertainty over the parametrized function. In this paper, we investigate this issue in the context of the increasingly popular linearized Laplace approximation. Specifically, it has been observed that linearized predictives alleviate the common underfitting problems of the Laplace approximation. We develop a new geometric view of reparametrizations from which we explain the success of linearization. Moreover, we demonstrate that these reparameterization invariance properties can be extended to the original neural network predictive using a Riemannian diffusion process giving a straightforward algorithm for approximate posterior sampling, which empirically improves posterior fit. |
| title | Reparameterization invariance in approximate Bayesian inference |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2406.03334 |