Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neurons
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915699920732160 |
|---|---|
| author | Dufort-Labbé, Simon D'Oro, Pierluca Nikishin, Evgenii Pascanu, Razvan Bacon, Pierre-Luc Baratin, Aristide |
| author_facet | Dufort-Labbé, Simon D'Oro, Pierluca Nikishin, Evgenii Pascanu, Razvan Bacon, Pierre-Luc Baratin, Aristide |
| contents | When training neural networks, dying neurons -- units becoming inactive or saturated -- are traditionally seen as harmful. This paper sheds new light on this phenomenon. By exploring the impact of various hyperparameter configurations on dying neurons during training, we gather insights on how to improve upon sparse training approaches to pruning. We introduce Demon Pruning (DemP), a method that controls the proliferation of dead neurons through a combination of noise injection on active units and a one-cycle schedule regularization strategy, dynamically leading to network sparsity. Experiments on CIFAR-10 and ImageNet datasets demonstrate that DemP outperforms existing dense-to-sparse structured pruning methods, achieving better accuracy-sparsity tradeoffs and accelerating training by up to 3.56$\times$. These findings provide a novel perspective on dying neurons as a resource for efficient model compression and optimization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_07688 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neurons Dufort-Labbé, Simon D'Oro, Pierluca Nikishin, Evgenii Pascanu, Razvan Bacon, Pierre-Luc Baratin, Aristide Machine Learning Artificial Intelligence When training neural networks, dying neurons -- units becoming inactive or saturated -- are traditionally seen as harmful. This paper sheds new light on this phenomenon. By exploring the impact of various hyperparameter configurations on dying neurons during training, we gather insights on how to improve upon sparse training approaches to pruning. We introduce Demon Pruning (DemP), a method that controls the proliferation of dead neurons through a combination of noise injection on active units and a one-cycle schedule regularization strategy, dynamically leading to network sparsity. Experiments on CIFAR-10 and ImageNet datasets demonstrate that DemP outperforms existing dense-to-sparse structured pruning methods, achieving better accuracy-sparsity tradeoffs and accelerating training by up to 3.56$\times$. These findings provide a novel perspective on dying neurons as a resource for efficient model compression and optimization. |
| title | Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neurons |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2403.07688 |