Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2310.03165 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914942231248896 |
|---|---|
| author | Berlyand, Leonid Sandier, Etienne Shmalo, Yitzchak Zhang, Lei |
| author_facet | Berlyand, Leonid Sandier, Etienne Shmalo, Yitzchak Zhang, Lei |
| contents | We explore the applications of random matrix theory (RMT) in the training of deep neural networks (DNNs), focusing on layer pruning that is reducing the number of DNN parameters (weights). Our numerical results show that this pruning leads to a drastic reduction of parameters while not reducing the accuracy of DNNs and CNNs. Moreover, pruning the fully connected DNNs actually increases the accuracy and decreases the variance for random initializations. Our numerics indicate that this enhancement in accuracy is due to the simplification of the loss landscape. We next provide rigorous mathematical underpinning of these numerical results by proving the RMT-based Pruning Theorem. Our results offer valuable insights into the practical application of RMT for the creation of more efficient and accurate deep-learning models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_03165 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Enhancing Accuracy in Deep Learning Using Random Matrix Theory Berlyand, Leonid Sandier, Etienne Shmalo, Yitzchak Zhang, Lei Machine Learning Optimization and Control We explore the applications of random matrix theory (RMT) in the training of deep neural networks (DNNs), focusing on layer pruning that is reducing the number of DNN parameters (weights). Our numerical results show that this pruning leads to a drastic reduction of parameters while not reducing the accuracy of DNNs and CNNs. Moreover, pruning the fully connected DNNs actually increases the accuracy and decreases the variance for random initializations. Our numerics indicate that this enhancement in accuracy is due to the simplification of the loss landscape. We next provide rigorous mathematical underpinning of these numerical results by proving the RMT-based Pruning Theorem. Our results offer valuable insights into the practical application of RMT for the creation of more efficient and accurate deep-learning models. |
| title | Enhancing Accuracy in Deep Learning Using Random Matrix Theory |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2310.03165 |