Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.10188 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917137999724544 |
|---|---|
| author | Clara, Gabriel Mash'al, Yazan |
| author_facet | Clara, Gabriel Mash'al, Yazan |
| contents | We analyze gradient descent with randomly weighted data points in a linear regression model, under a generic weighting distribution. This includes various forms of stochastic gradient descent, importance sampling, but also extends to weighting distributions with arbitrary continuous values, thereby providing a unified framework to analyze the impact of various kinds of noise on the training trajectory. We characterize the implicit regularization induced through the random weighting, connect it with weighted linear regression, and derive non-asymptotic bounds for convergence in first and second moments. Leveraging geometric moment contraction, we also investigate the stationary distribution induced by the added noise. Based on these results, we discuss how specific choices of weighting distribution influence both the underlying optimization problem and statistical properties of the resulting estimator, as well as some examples for which weightings that lead to fast convergence cause bad statistical performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_10188 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The Interplay of Statistics and Noisy Optimization: Learning Linear Predictors with Random Data Weights Clara, Gabriel Mash'al, Yazan Machine Learning Computation We analyze gradient descent with randomly weighted data points in a linear regression model, under a generic weighting distribution. This includes various forms of stochastic gradient descent, importance sampling, but also extends to weighting distributions with arbitrary continuous values, thereby providing a unified framework to analyze the impact of various kinds of noise on the training trajectory. We characterize the implicit regularization induced through the random weighting, connect it with weighted linear regression, and derive non-asymptotic bounds for convergence in first and second moments. Leveraging geometric moment contraction, we also investigate the stationary distribution induced by the added noise. Based on these results, we discuss how specific choices of weighting distribution influence both the underlying optimization problem and statistical properties of the resulting estimator, as well as some examples for which weightings that lead to fast convergence cause bad statistical performance. |
| title | The Interplay of Statistics and Noisy Optimization: Learning Linear Predictors with Random Data Weights |
| topic | Machine Learning Computation |
| url | https://arxiv.org/abs/2512.10188 |