Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.16734 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915512522375168 |
|---|---|
| author | Feoktistov, Dmitrii Ignashin, Igor Veprikov, Andrey Borovko, Nikita Bogdanov, Alexander Chezhegov, Savelii Beznosikov, Aleksandr |
| author_facet | Feoktistov, Dmitrii Ignashin, Igor Veprikov, Andrey Borovko, Nikita Bogdanov, Alexander Chezhegov, Savelii Beznosikov, Aleksandr |
| contents | While traditional Deep Learning (DL) optimization methods treat all training samples equally, Distributionally Robust Optimization (DRO) adaptively assigns importance weights to different samples. However, a significant gap exists between DRO and current DL practices. Modern DL optimizers require adaptivity and the ability to handle stochastic gradients, as these methods demonstrate superior performance. Additionally, for practical applications, a method should allow weight assignment not only to individual samples, but also to groups of objects (for example, all samples of the same class). This paper aims to bridge this gap by introducing ALSO $\unicode{x2013}$ Adaptive Loss Scaling Optimizer $\unicode{x2013}$ an adaptive algorithm for a modified DRO objective that can handle weight assignment to sample groups. We prove the convergence of our proposed algorithm for non-convex objectives, which is the typical case for DL models. Empirical evaluation across diverse Deep Learning tasks, from Tabular DL to Split Learning tasks, demonstrates that ALSO outperforms both traditional optimizers and existing DRO methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_16734 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Aligning Distributionally Robust Optimization with Practical Deep Learning Needs Feoktistov, Dmitrii Ignashin, Igor Veprikov, Andrey Borovko, Nikita Bogdanov, Alexander Chezhegov, Savelii Beznosikov, Aleksandr Machine Learning While traditional Deep Learning (DL) optimization methods treat all training samples equally, Distributionally Robust Optimization (DRO) adaptively assigns importance weights to different samples. However, a significant gap exists between DRO and current DL practices. Modern DL optimizers require adaptivity and the ability to handle stochastic gradients, as these methods demonstrate superior performance. Additionally, for practical applications, a method should allow weight assignment not only to individual samples, but also to groups of objects (for example, all samples of the same class). This paper aims to bridge this gap by introducing ALSO $\unicode{x2013}$ Adaptive Loss Scaling Optimizer $\unicode{x2013}$ an adaptive algorithm for a modified DRO objective that can handle weight assignment to sample groups. We prove the convergence of our proposed algorithm for non-convex objectives, which is the typical case for DL models. Empirical evaluation across diverse Deep Learning tasks, from Tabular DL to Split Learning tasks, demonstrates that ALSO outperforms both traditional optimizers and existing DRO methods. |
| title | Aligning Distributionally Robust Optimization with Practical Deep Learning Needs |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2508.16734 |