Graph Expansion in Pruned Recurrent Neural Network Layers Preserve Performance
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909139296321536 |
|---|---|
| author | Kalra, Suryam Arnav Biswas, Arindam Mitra, Pabitra Basu, Biswajit |
| author_facet | Kalra, Suryam Arnav Biswas, Arindam Mitra, Pabitra Basu, Biswajit |
| contents | Expansion property of a graph refers to its strong connectivity as well as sparseness. It has been reported that deep neural networks can be pruned to a high degree of sparsity while maintaining their performance. Such pruning is essential for performing real time sequence learning tasks using recurrent neural networks in resource constrained platforms. We prune recurrent networks such as RNNs and LSTMs, maintaining a large spectral gap of the underlying graphs and ensuring their layerwise expansion properties. We also study the time unfolded recurrent network graphs in terms of the properties of their bipartite layers. Experimental results for the benchmark sequence MNIST, CIFAR-10, and Google speech command data show that expander graph properties are key to preserving classification accuracy of RNN and LSTM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_11100 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Graph Expansion in Pruned Recurrent Neural Network Layers Preserve Performance Kalra, Suryam Arnav Biswas, Arindam Mitra, Pabitra Basu, Biswajit Machine Learning Computer Vision and Pattern Recognition Neural and Evolutionary Computing 05C68 I.2.6 Expansion property of a graph refers to its strong connectivity as well as sparseness. It has been reported that deep neural networks can be pruned to a high degree of sparsity while maintaining their performance. Such pruning is essential for performing real time sequence learning tasks using recurrent neural networks in resource constrained platforms. We prune recurrent networks such as RNNs and LSTMs, maintaining a large spectral gap of the underlying graphs and ensuring their layerwise expansion properties. We also study the time unfolded recurrent network graphs in terms of the properties of their bipartite layers. Experimental results for the benchmark sequence MNIST, CIFAR-10, and Google speech command data show that expander graph properties are key to preserving classification accuracy of RNN and LSTM. |
| title | Graph Expansion in Pruned Recurrent Neural Network Layers Preserve Performance |
| topic | Machine Learning Computer Vision and Pattern Recognition Neural and Evolutionary Computing 05C68 I.2.6 |
| url | https://arxiv.org/abs/2403.11100 |