Learning Neural Network Classifiers with Low Model Complexity
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2017
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912090950729728 |
|---|---|
| author | Jayadeva Pant, Himanshu Sharma, Mayank Dubey, Abhimanyu Soman, Sumit Tripathi, Suraj Guruju, Sai Goalla, Nihal |
| author_facet | Jayadeva Pant, Himanshu Sharma, Mayank Dubey, Abhimanyu Soman, Sumit Tripathi, Suraj Guruju, Sai Goalla, Nihal |
| contents | Modern neural network architectures for large-scale learning tasks have substantially higher model complexities, which makes understanding, visualizing and training these architectures difficult. Recent contributions to deep learning techniques have focused on architectural modifications to improve parameter efficiency and performance. In this paper, we derive a continuous and differentiable error functional for a neural network that minimizes its empirical error as well as a measure of the model complexity. The latter measure is obtained by deriving a differentiable upper bound on the Vapnik-Chervonenkis (VC) dimension of the classifier layer of a class of deep networks. Using standard backpropagation, we realize a training rule that tries to minimize the error on training samples, while improving generalization by keeping the model complexity low. We demonstrate the effectiveness of our formulation (the Low Complexity Neural Network - LCNN) across several deep learning algorithms, and a variety of large benchmark datasets. We show that hidden layer neurons in the resultant networks learn features that are crisp, and in the case of image datasets, quantitatively sharper. Our proposed approach yields benefits across a wide range of architectures, in comparison to and in conjunction with methods such as Dropout and Batch Normalization, and our results strongly suggest that deep learning techniques can benefit from model complexity control methods such as the LCNN learning rule. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_1707_09933 |
| institution | arXiv |
| publishDate | 2017 |
| record_format | arxiv |
| spellingShingle | Learning Neural Network Classifiers with Low Model Complexity Jayadeva Pant, Himanshu Sharma, Mayank Dubey, Abhimanyu Soman, Sumit Tripathi, Suraj Guruju, Sai Goalla, Nihal Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition 68T05, 68T10, 68Q32 Modern neural network architectures for large-scale learning tasks have substantially higher model complexities, which makes understanding, visualizing and training these architectures difficult. Recent contributions to deep learning techniques have focused on architectural modifications to improve parameter efficiency and performance. In this paper, we derive a continuous and differentiable error functional for a neural network that minimizes its empirical error as well as a measure of the model complexity. The latter measure is obtained by deriving a differentiable upper bound on the Vapnik-Chervonenkis (VC) dimension of the classifier layer of a class of deep networks. Using standard backpropagation, we realize a training rule that tries to minimize the error on training samples, while improving generalization by keeping the model complexity low. We demonstrate the effectiveness of our formulation (the Low Complexity Neural Network - LCNN) across several deep learning algorithms, and a variety of large benchmark datasets. We show that hidden layer neurons in the resultant networks learn features that are crisp, and in the case of image datasets, quantitatively sharper. Our proposed approach yields benefits across a wide range of architectures, in comparison to and in conjunction with methods such as Dropout and Batch Normalization, and our results strongly suggest that deep learning techniques can benefit from model complexity control methods such as the LCNN learning rule. |
| title | Learning Neural Network Classifiers with Low Model Complexity |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition 68T05, 68T10, 68Q32 |
| url | https://arxiv.org/abs/1707.09933 |