Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2410.22065 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909370572341248 |
|---|---|
| author | Dinh, Vu C. Ho, Lam Si Tung Nguyen, Cuong V. |
| author_facet | Dinh, Vu C. Ho, Lam Si Tung Nguyen, Cuong V. |
| contents | We analyze the error rates of the Hamiltonian Monte Carlo algorithm with leapfrog integrator for Bayesian neural network inference. We show that due to the non-differentiability of activation functions in the ReLU family, leapfrog HMC for networks with these activation functions has a large local error rate of $Ω(ε)$ rather than the classical error rate of $O(ε^3)$. This leads to a higher rejection rate of the proposals, making the method inefficient. We then verify our theoretical findings through empirical simulations as well as experiments on a real-world dataset that highlight the inefficiency of HMC inference on ReLU-based neural networks compared to analytical networks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_22065 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Hamiltonian Monte Carlo on ReLU Neural Networks is Inefficient Dinh, Vu C. Ho, Lam Si Tung Nguyen, Cuong V. Machine Learning We analyze the error rates of the Hamiltonian Monte Carlo algorithm with leapfrog integrator for Bayesian neural network inference. We show that due to the non-differentiability of activation functions in the ReLU family, leapfrog HMC for networks with these activation functions has a large local error rate of $Ω(ε)$ rather than the classical error rate of $O(ε^3)$. This leads to a higher rejection rate of the proposals, making the method inefficient. We then verify our theoretical findings through empirical simulations as well as experiments on a real-world dataset that highlight the inefficiency of HMC inference on ReLU-based neural networks compared to analytical networks. |
| title | Hamiltonian Monte Carlo on ReLU Neural Networks is Inefficient |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2410.22065 |