Hamiltonian Monte Carlo on ReLU Neural Networks is Inefficient
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909370572341248 |
|---|---|
| author | Dinh, Vu C. Ho, Lam Si Tung Nguyen, Cuong V. |
| author_facet | Dinh, Vu C. Ho, Lam Si Tung Nguyen, Cuong V. |
| contents | We analyze the error rates of the Hamiltonian Monte Carlo algorithm with leapfrog integrator for Bayesian neural network inference. We show that due to the non-differentiability of activation functions in the ReLU family, leapfrog HMC for networks with these activation functions has a large local error rate of $Ω(ε)$ rather than the classical error rate of $O(ε^3)$. This leads to a higher rejection rate of the proposals, making the method inefficient. We then verify our theoretical findings through empirical simulations as well as experiments on a real-world dataset that highlight the inefficiency of HMC inference on ReLU-based neural networks compared to analytical networks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_22065 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Hamiltonian Monte Carlo on ReLU Neural Networks is Inefficient Dinh, Vu C. Ho, Lam Si Tung Nguyen, Cuong V. Machine Learning We analyze the error rates of the Hamiltonian Monte Carlo algorithm with leapfrog integrator for Bayesian neural network inference. We show that due to the non-differentiability of activation functions in the ReLU family, leapfrog HMC for networks with these activation functions has a large local error rate of $Ω(ε)$ rather than the classical error rate of $O(ε^3)$. This leads to a higher rejection rate of the proposals, making the method inefficient. We then verify our theoretical findings through empirical simulations as well as experiments on a real-world dataset that highlight the inefficiency of HMC inference on ReLU-based neural networks compared to analytical networks. |
| title | Hamiltonian Monte Carlo on ReLU Neural Networks is Inefficient |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2410.22065 |