Sparse Training of Neural Networks based on Multilevel Mirror Descent
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913139139805184 |
|---|---|
| author | Lunk, Yannick Scott, Sebastian J. Bungert, Leon |
| author_facet | Lunk, Yannick Scott, Sebastian J. Bungert, Leon |
| contents | We introduce a dynamic sparse training algorithm based on linearized Bregman iterations / mirror descent that exploits the naturally incurred sparsity by alternating between periods of static and dynamic sparsity pattern updates. The key idea is to combine sparsity-inducing Bregman iterations with adaptive freezing of the network structure to enable efficient exploration of the sparse parameter space while maintaining sparsity. We provide convergence guaranties by embedding our method in a multilevel optimization framework. Furthermore, we empirically show that our algorithm can produce highly sparse and accurate models on standard benchmarks. We also show that the theoretical number of FLOPs compared to SGD training can be reduced from 38% for standard Bregman iterations to 6% for our method while maintaining test accuracy.We additionally show a training time reduction by about 50%, when using a sparsity-aware CPU implementation of our method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_03535 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Sparse Training of Neural Networks based on Multilevel Mirror Descent Lunk, Yannick Scott, Sebastian J. Bungert, Leon Machine Learning Numerical Analysis Optimization and Control 65K10, 68T05, 90C26 I.2.6; F.2.1; G.1.6 We introduce a dynamic sparse training algorithm based on linearized Bregman iterations / mirror descent that exploits the naturally incurred sparsity by alternating between periods of static and dynamic sparsity pattern updates. The key idea is to combine sparsity-inducing Bregman iterations with adaptive freezing of the network structure to enable efficient exploration of the sparse parameter space while maintaining sparsity. We provide convergence guaranties by embedding our method in a multilevel optimization framework. Furthermore, we empirically show that our algorithm can produce highly sparse and accurate models on standard benchmarks. We also show that the theoretical number of FLOPs compared to SGD training can be reduced from 38% for standard Bregman iterations to 6% for our method while maintaining test accuracy.We additionally show a training time reduction by about 50%, when using a sparsity-aware CPU implementation of our method. |
| title | Sparse Training of Neural Networks based on Multilevel Mirror Descent |
| topic | Machine Learning Numerical Analysis Optimization and Control 65K10, 68T05, 90C26 I.2.6; F.2.1; G.1.6 |
| url | https://arxiv.org/abs/2602.03535 |