Stochastic Re-weighted Gradient Descent via Distributionally Robust Optimization
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917801484091392 |
|---|---|
| author | Kumar, Ramnath Majmundar, Kushal Nagaraj, Dheeraj Suggala, Arun Sai |
| author_facet | Kumar, Ramnath Majmundar, Kushal Nagaraj, Dheeraj Suggala, Arun Sai |
| contents | We present Re-weighted Gradient Descent (RGD), a novel optimization technique that improves the performance of deep neural networks through dynamic sample re-weighting. Leveraging insights from distributionally robust optimization (DRO) with Kullback-Leibler divergence, our method dynamically assigns importance weights to training data during each optimization step. RGD is simple to implement, computationally efficient, and compatible with widely used optimizers such as SGD and Adam. We demonstrate the effectiveness of RGD on various learning tasks, including supervised learning, meta-learning, and out-of-domain generalization. Notably, RGD achieves state-of-the-art results on diverse benchmarks, with improvements of +0.7% on DomainBed, +1.44% on tabular classification, \textcolor{blue}+1.94% on GLUE with BERT, and +1.01% on ImageNet-1K with ViT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2306_09222 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Stochastic Re-weighted Gradient Descent via Distributionally Robust Optimization Kumar, Ramnath Majmundar, Kushal Nagaraj, Dheeraj Suggala, Arun Sai Machine Learning Artificial Intelligence We present Re-weighted Gradient Descent (RGD), a novel optimization technique that improves the performance of deep neural networks through dynamic sample re-weighting. Leveraging insights from distributionally robust optimization (DRO) with Kullback-Leibler divergence, our method dynamically assigns importance weights to training data during each optimization step. RGD is simple to implement, computationally efficient, and compatible with widely used optimizers such as SGD and Adam. We demonstrate the effectiveness of RGD on various learning tasks, including supervised learning, meta-learning, and out-of-domain generalization. Notably, RGD achieves state-of-the-art results on diverse benchmarks, with improvements of +0.7% on DomainBed, +1.44% on tabular classification, \textcolor{blue}+1.94% on GLUE with BERT, and +1.01% on ImageNet-1K with ViT. |
| title | Stochastic Re-weighted Gradient Descent via Distributionally Robust Optimization |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2306.09222 |