Refining Covariance Matrix Estimation in Stochastic Gradient Descent Through Bias Reduction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913056626311168 |
|---|---|
| author | Wei, Ziyang Zhu, Wanrong Lyu, Jingyang Wu, Wei Biao |
| author_facet | Wei, Ziyang Zhu, Wanrong Lyu, Jingyang Wu, Wei Biao |
| contents | We study online inference and asymptotic covariance estimation for the stochastic gradient descent (SGD) algorithm. While classical methods (such as plug-in and batch-means estimators) are available, they either require inaccessible second-order (Hessian) information or suffer from slow convergence. To address these challenges, we propose a novel, fully online de-biased covariance estimator that eliminates the need for second-order derivatives while significantly improving estimation accuracy. Our method employs a bias-reduction technique to achieve a convergence rate of $n^{(α-1)/2} \sqrt{\log n}$, outperforming existing Hessian-free alternatives. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_21203 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Refining Covariance Matrix Estimation in Stochastic Gradient Descent Through Bias Reduction Wei, Ziyang Zhu, Wanrong Lyu, Jingyang Wu, Wei Biao Machine Learning We study online inference and asymptotic covariance estimation for the stochastic gradient descent (SGD) algorithm. While classical methods (such as plug-in and batch-means estimators) are available, they either require inaccessible second-order (Hessian) information or suffer from slow convergence. To address these challenges, we propose a novel, fully online de-biased covariance estimator that eliminates the need for second-order derivatives while significantly improving estimation accuracy. Our method employs a bias-reduction technique to achieve a convergence rate of $n^{(α-1)/2} \sqrt{\log n}$, outperforming existing Hessian-free alternatives. |
| title | Refining Covariance Matrix Estimation in Stochastic Gradient Descent Through Bias Reduction |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2604.21203 |