On the Batch Size Selection in Stochastic Gradient Methods Using No-Replacement Sampling
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916788212596736 |
|---|---|
| author | Boresta, Marco De Santis, Alberto Lucidi, Stefano |
| author_facet | Boresta, Marco De Santis, Alberto Lucidi, Stefano |
| contents | Recent stochastic gradient methods that have appeared in the literature base their efficiency and global convergence properties on a suitable control of the variance of the gradient batch estimate. This control is typically achieved by dynamically increasing the batch size during the iterations of the algorithm. However, in the existing methods the statistical analysis often relies on sampling with replacement. This particular batch selection appears unrealistic in practice. In this paper, we consider a more realistic approach to batch size selection based on sampling without replacement. The consequent statistical analysis is compared to that of sampling with replacement. The new batch size selection method, while still ensuring global convergence, provides a more accurate representation of the variance reduction observed in practice, leading to a smoother and more efficient batch size update scheme. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_08758 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | On the Batch Size Selection in Stochastic Gradient Methods Using No-Replacement Sampling Boresta, Marco De Santis, Alberto Lucidi, Stefano Optimization and Control Recent stochastic gradient methods that have appeared in the literature base their efficiency and global convergence properties on a suitable control of the variance of the gradient batch estimate. This control is typically achieved by dynamically increasing the batch size during the iterations of the algorithm. However, in the existing methods the statistical analysis often relies on sampling with replacement. This particular batch selection appears unrealistic in practice. In this paper, we consider a more realistic approach to batch size selection based on sampling without replacement. The consequent statistical analysis is compared to that of sampling with replacement. The new batch size selection method, while still ensuring global convergence, provides a more accurate representation of the variance reduction observed in practice, leading to a smoother and more efficient batch size update scheme. |
| title | On the Batch Size Selection in Stochastic Gradient Methods Using No-Replacement Sampling |
| topic | Optimization and Control |
| url | https://arxiv.org/abs/2506.08758 |