Rényi's $α$-divergence variational Bayes for spike-and-slab high-dimensional linear regression
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911294742855680 |
|---|---|
| author | Bsila, Chadi Tang, Yiqi Wang, Kaiwen Heyer, Laurie |
| author_facet | Bsila, Chadi Tang, Yiqi Wang, Kaiwen Heyer, Laurie |
| contents | Sparse high-dimensional linear regression is a central problem in statistics, where the goal is often variable selection and/or coefficient estimation. We propose a mean-field variational Bayes approximation for sparse regression with spike-and-slab Laplace priors that replaces the standard Kullback-Leibler (KL) divergence objective with the Rényi's $α$ divergence: a one-parameter generalization of KL divergence indexed by $α\in (0, \infty) \setminus \{1\}$ that allows flexibility between zero-forcing and mass-covering behavior. We derive coordinate ascent variational inference (CAVI) updates via a second-order delta method and develop a stochastic variational inference algorithm based on a Monte Carlo surrogate Rényi lower bound. In simulations, our two methods perform comparably to state-of-the-art Bayesian variable selection procedures across a range of sparsity configurations and $α$ values for both variable selection and estimation, and our numerical results illustrate how different choices of $α$ can be advantageous in different sparsity configurations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_00627 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Rényi's $α$-divergence variational Bayes for spike-and-slab high-dimensional linear regression Bsila, Chadi Tang, Yiqi Wang, Kaiwen Heyer, Laurie Methodology Computation Sparse high-dimensional linear regression is a central problem in statistics, where the goal is often variable selection and/or coefficient estimation. We propose a mean-field variational Bayes approximation for sparse regression with spike-and-slab Laplace priors that replaces the standard Kullback-Leibler (KL) divergence objective with the Rényi's $α$ divergence: a one-parameter generalization of KL divergence indexed by $α\in (0, \infty) \setminus \{1\}$ that allows flexibility between zero-forcing and mass-covering behavior. We derive coordinate ascent variational inference (CAVI) updates via a second-order delta method and develop a stochastic variational inference algorithm based on a Monte Carlo surrogate Rényi lower bound. In simulations, our two methods perform comparably to state-of-the-art Bayesian variable selection procedures across a range of sparsity configurations and $α$ values for both variable selection and estimation, and our numerical results illustrate how different choices of $α$ can be advantageous in different sparsity configurations. |
| title | Rényi's $α$-divergence variational Bayes for spike-and-slab high-dimensional linear regression |
| topic | Methodology Computation |
| url | https://arxiv.org/abs/2512.00627 |