COLT: Cyclic Overlapping Lottery Tickets for Faster Pruning of Convolutional Neural Networks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2022
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910796537135104 |
|---|---|
| author | Hossain, Md. Ismail Rakib, Mohammed Elahi, M. M. Lutfe Mohammed, Nabeel Rahman, Shafin |
| author_facet | Hossain, Md. Ismail Rakib, Mohammed Elahi, M. M. Lutfe Mohammed, Nabeel Rahman, Shafin |
| contents | Pruning refers to the elimination of trivial weights from neural networks. The sub-networks within an overparameterized model produced after pruning are often called Lottery tickets. This research aims to generate winning lottery tickets from a set of lottery tickets that can achieve similar accuracy to the original unpruned network. We introduce a novel winning ticket called Cyclic Overlapping Lottery Ticket (COLT) by data splitting and cyclic retraining of the pruned network from scratch. We apply a cyclic pruning algorithm that keeps only the overlapping weights of different pruned models trained on different data segments. Our results demonstrate that COLT can achieve similar accuracies (obtained by the unpruned model) while maintaining high sparsities. We show that the accuracy of COLT is on par with the winning tickets of Lottery Ticket Hypothesis (LTH) and, at times, is better. Moreover, COLTs can be generated using fewer iterations than tickets generated by the popular Iterative Magnitude Pruning (IMP) method. In addition, we also notice COLTs generated on large datasets can be transferred to small ones without compromising performance, demonstrating its generalizing capability. We conduct all our experiments on Cifar-10, Cifar-100 & TinyImageNet datasets and report superior performance than the state-of-the-art methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2212_12770 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | COLT: Cyclic Overlapping Lottery Tickets for Faster Pruning of Convolutional Neural Networks Hossain, Md. Ismail Rakib, Mohammed Elahi, M. M. Lutfe Mohammed, Nabeel Rahman, Shafin Computer Vision and Pattern Recognition Pruning refers to the elimination of trivial weights from neural networks. The sub-networks within an overparameterized model produced after pruning are often called Lottery tickets. This research aims to generate winning lottery tickets from a set of lottery tickets that can achieve similar accuracy to the original unpruned network. We introduce a novel winning ticket called Cyclic Overlapping Lottery Ticket (COLT) by data splitting and cyclic retraining of the pruned network from scratch. We apply a cyclic pruning algorithm that keeps only the overlapping weights of different pruned models trained on different data segments. Our results demonstrate that COLT can achieve similar accuracies (obtained by the unpruned model) while maintaining high sparsities. We show that the accuracy of COLT is on par with the winning tickets of Lottery Ticket Hypothesis (LTH) and, at times, is better. Moreover, COLTs can be generated using fewer iterations than tickets generated by the popular Iterative Magnitude Pruning (IMP) method. In addition, we also notice COLTs generated on large datasets can be transferred to small ones without compromising performance, demonstrating its generalizing capability. We conduct all our experiments on Cifar-10, Cifar-100 & TinyImageNet datasets and report superior performance than the state-of-the-art methods. |
| title | COLT: Cyclic Overlapping Lottery Tickets for Faster Pruning of Convolutional Neural Networks |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2212.12770 |