Comparison of parallel SMC and MCMC for Bayesian deep learning
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911111738032128 |
|---|---|
| author | Liang, Xinzhu Lukens, Joseph M. Lohani, Sanjaya Kirby, Brian T. Searles, Thomas A. Qiu, Xin Law, Kody J. H. |
| author_facet | Liang, Xinzhu Lukens, Joseph M. Lohani, Sanjaya Kirby, Brian T. Searles, Thomas A. Qiu, Xin Law, Kody J. H. |
| contents | This work systematically compares parallel implementations of consistent (asymptotically unbiased) Bayesian deep learning algorithms: sequential Monte Carlo sampler (SMC$_\parallel$) or Markov chain Monte Carlo (MCMC$_\parallel$). We provide a proof of convergence for SMC$_\parallel$ showing that it theoretically achieves the same level of convergence as a single monolithic SMC sampler, while the reduced communication lowers wall-clock time. It is well-known that the first samples from MCMC need to be discarded to eliminate initialization bias, and that the number of discarded samples must grow like the logarithm of the number of parallel chains to control that bias for MCMC$_\parallel$. A systematic empirical numerical study on MNIST, CIFAR, and IMDb, reveals that parallel implementations of both methods perform comparably to non-parallel implementations in terms of performance and total cost, and also comparably to each other. However, both methods still require a large wall-clock time, and suffer from catastrophic non-convergence if they aren't run for long enough. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_06173 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Comparison of parallel SMC and MCMC for Bayesian deep learning Liang, Xinzhu Lukens, Joseph M. Lohani, Sanjaya Kirby, Brian T. Searles, Thomas A. Qiu, Xin Law, Kody J. H. Machine Learning Computation This work systematically compares parallel implementations of consistent (asymptotically unbiased) Bayesian deep learning algorithms: sequential Monte Carlo sampler (SMC$_\parallel$) or Markov chain Monte Carlo (MCMC$_\parallel$). We provide a proof of convergence for SMC$_\parallel$ showing that it theoretically achieves the same level of convergence as a single monolithic SMC sampler, while the reduced communication lowers wall-clock time. It is well-known that the first samples from MCMC need to be discarded to eliminate initialization bias, and that the number of discarded samples must grow like the logarithm of the number of parallel chains to control that bias for MCMC$_\parallel$. A systematic empirical numerical study on MNIST, CIFAR, and IMDb, reveals that parallel implementations of both methods perform comparably to non-parallel implementations in terms of performance and total cost, and also comparably to each other. However, both methods still require a large wall-clock time, and suffer from catastrophic non-convergence if they aren't run for long enough. |
| title | Comparison of parallel SMC and MCMC for Bayesian deep learning |
| topic | Machine Learning Computation |
| url | https://arxiv.org/abs/2402.06173 |