Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.15525 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909467949400064 |
|---|---|
| author | Salaün, Corentin Huang, Xingchang Georgiev, Iliyan Mitra, Niloy J. Singh, Gurprit |
| author_facet | Salaün, Corentin Huang, Xingchang Georgiev, Iliyan Mitra, Niloy J. Singh, Gurprit |
| contents | We introduce a theoretical and practical framework for efficient importance sampling of mini-batch samples for gradient estimation from single and multiple probability distributions. To handle noisy gradients, our framework dynamically evolves the importance distribution during training by utilizing a self-adaptive metric. Our framework combines multiple, diverse sampling distributions, each tailored to specific parameter gradients. This approach facilitates the importance sampling of vector-valued gradient estimation. Rather than naively combining multiple distributions, our framework involves optimally weighting data contribution across multiple distributions. This adapted combination of multiple importance yields superior gradient estimates, leading to faster training convergence. We demonstrate the effectiveness of our approach through empirical evaluations across a range of optimization tasks like classification and regression on both image and point cloud datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_15525 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multiple Importance Sampling for Stochastic Gradient Estimation Salaün, Corentin Huang, Xingchang Georgiev, Iliyan Mitra, Niloy J. Singh, Gurprit Machine Learning We introduce a theoretical and practical framework for efficient importance sampling of mini-batch samples for gradient estimation from single and multiple probability distributions. To handle noisy gradients, our framework dynamically evolves the importance distribution during training by utilizing a self-adaptive metric. Our framework combines multiple, diverse sampling distributions, each tailored to specific parameter gradients. This approach facilitates the importance sampling of vector-valued gradient estimation. Rather than naively combining multiple distributions, our framework involves optimally weighting data contribution across multiple distributions. This adapted combination of multiple importance yields superior gradient estimates, leading to faster training convergence. We demonstrate the effectiveness of our approach through empirical evaluations across a range of optimization tasks like classification and regression on both image and point cloud datasets. |
| title | Multiple Importance Sampling for Stochastic Gradient Estimation |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2407.15525 |