Pruning By Explaining Revisited: Optimizing Attribution Methods to Prune CNNs and Transformers
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912082902908928 |
|---|---|
| author | Hatefi, Sayed Mohammad Vakilzadeh Dreyer, Maximilian Achtibat, Reduan Wiegand, Thomas Samek, Wojciech Lapuschkin, Sebastian |
| author_facet | Hatefi, Sayed Mohammad Vakilzadeh Dreyer, Maximilian Achtibat, Reduan Wiegand, Thomas Samek, Wojciech Lapuschkin, Sebastian |
| contents | To solve ever more complex problems, Deep Neural Networks are scaled to billions of parameters, leading to huge computational costs. An effective approach to reduce computational requirements and increase efficiency is to prune unnecessary components of these often over-parameterized networks. Previous work has shown that attribution methods from the field of eXplainable AI serve as effective means to extract and prune the least relevant network components in a few-shot fashion. We extend the current state by proposing to explicitly optimize hyperparameters of attribution methods for the task of pruning, and further include transformer-based networks in our analysis. Our approach yields higher model compression rates of large transformer- and convolutional architectures (VGG, ResNet, ViT) compared to previous works, while still attaining high performance on ImageNet classification tasks. Here, our experiments indicate that transformers have a higher degree of over-parameterization compared to convolutional neural networks. Code is available at https://github.com/erfanhatefi/Pruning-by-eXplaining-in-PyTorch. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_12568 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Pruning By Explaining Revisited: Optimizing Attribution Methods to Prune CNNs and Transformers Hatefi, Sayed Mohammad Vakilzadeh Dreyer, Maximilian Achtibat, Reduan Wiegand, Thomas Samek, Wojciech Lapuschkin, Sebastian Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning To solve ever more complex problems, Deep Neural Networks are scaled to billions of parameters, leading to huge computational costs. An effective approach to reduce computational requirements and increase efficiency is to prune unnecessary components of these often over-parameterized networks. Previous work has shown that attribution methods from the field of eXplainable AI serve as effective means to extract and prune the least relevant network components in a few-shot fashion. We extend the current state by proposing to explicitly optimize hyperparameters of attribution methods for the task of pruning, and further include transformer-based networks in our analysis. Our approach yields higher model compression rates of large transformer- and convolutional architectures (VGG, ResNet, ViT) compared to previous works, while still attaining high performance on ImageNet classification tasks. Here, our experiments indicate that transformers have a higher degree of over-parameterization compared to convolutional neural networks. Code is available at https://github.com/erfanhatefi/Pruning-by-eXplaining-in-PyTorch. |
| title | Pruning By Explaining Revisited: Optimizing Attribution Methods to Prune CNNs and Transformers |
| topic | Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2408.12568 |