Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.13482 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916378908295168 |
|---|---|
| author | Hu, Seungbeom Park, ChanJun Ferraiuolo, Andrew Ko, Sang-Ki Kim, Jinwoo Song, Haein Kim, Jieung |
| author_facet | Hu, Seungbeom Park, ChanJun Ferraiuolo, Andrew Ko, Sang-Ki Kim, Jinwoo Song, Haein Kim, Jieung |
| contents | Determining the optimal size of a neural network is critical, as it directly impacts runtime performance and memory usage. Pruning is a well-established model compression technique that reduces the size of neural networks while mathematically guaranteeing accuracy preservation. However, many recent pruning methods overlook the global contributions of individual model components, making it difficult to ensure that a pruned model meets the desired dataset and performance requirements. To address these challenges, we developed a new pruning algorithm, MPruner, that leverages mutual information through vector similarity. MPruner utilizes layer clustering with the Centered Kernel Alignment (CKA) similarity metric, allowing us to incorporate global information from the neural network for more precise and efficient layer-wise pruning. We evaluated MPruner across various architectures and configurations, demonstrating its versatility and providing practical guidelines. MPruner achieved up to a 50% reduction in parameters and memory usage for CNN and transformer-based models, with minimal to no loss in accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_13482 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MPruner: Optimizing Neural Network Size with CKA-Based Mutual Information Pruning Hu, Seungbeom Park, ChanJun Ferraiuolo, Andrew Ko, Sang-Ki Kim, Jinwoo Song, Haein Kim, Jieung Machine Learning Artificial Intelligence Determining the optimal size of a neural network is critical, as it directly impacts runtime performance and memory usage. Pruning is a well-established model compression technique that reduces the size of neural networks while mathematically guaranteeing accuracy preservation. However, many recent pruning methods overlook the global contributions of individual model components, making it difficult to ensure that a pruned model meets the desired dataset and performance requirements. To address these challenges, we developed a new pruning algorithm, MPruner, that leverages mutual information through vector similarity. MPruner utilizes layer clustering with the Centered Kernel Alignment (CKA) similarity metric, allowing us to incorporate global information from the neural network for more precise and efficient layer-wise pruning. We evaluated MPruner across various architectures and configurations, demonstrating its versatility and providing practical guidelines. MPruner achieved up to a 50% reduction in parameters and memory usage for CNN and transformer-based models, with minimal to no loss in accuracy. |
| title | MPruner: Optimizing Neural Network Size with CKA-Based Mutual Information Pruning |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2408.13482 |