SVRG and Beyond via Posterior Correction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911296596738048 |
|---|---|
| author | Daheim, Nico Möllenhoff, Thomas Ang, Ming Liang Khan, Mohammad Emtiyaz |
| author_facet | Daheim, Nico Möllenhoff, Thomas Ang, Ming Liang Khan, Mohammad Emtiyaz |
| contents | Stochastic Variance Reduced Gradient (SVRG) and its variants aim to speed-up training by using gradient corrections, but have seen limited success in deep learning. Here, we show surprising new foundational connections of SVRG to a recently proposed Bayesian method called posterior correction. Specifically, we show that SVRG is recovered as a special case of posterior correction over the isotropic-Gaussian family, while novel extensions are automatically obtained by using more flexible exponential families. We derive two new SVRG variants by using Gaussian families: First, a Newton-like variant that employs novel Hessian corrections, and second, an Adam-like extension that improves pretraining and finetuning of Transformer language models. This is the first work to connect SVRG to Bayes and use it to boost variational training for deep networks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_01930 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SVRG and Beyond via Posterior Correction Daheim, Nico Möllenhoff, Thomas Ang, Ming Liang Khan, Mohammad Emtiyaz Machine Learning Artificial Intelligence Stochastic Variance Reduced Gradient (SVRG) and its variants aim to speed-up training by using gradient corrections, but have seen limited success in deep learning. Here, we show surprising new foundational connections of SVRG to a recently proposed Bayesian method called posterior correction. Specifically, we show that SVRG is recovered as a special case of posterior correction over the isotropic-Gaussian family, while novel extensions are automatically obtained by using more flexible exponential families. We derive two new SVRG variants by using Gaussian families: First, a Newton-like variant that employs novel Hessian corrections, and second, an Adam-like extension that improves pretraining and finetuning of Transformer language models. This is the first work to connect SVRG to Bayes and use it to boost variational training for deep networks. |
| title | SVRG and Beyond via Posterior Correction |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2512.01930 |