Saved in:
| Main Author: | |
|---|---|
| Format: | Preprint |
| Published: |
2021
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2112.09741 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910560551960576 |
|---|---|
| author | Su, Weijie J. |
| author_facet | Su, Weijie J. |
| contents | To advance deep learning methodologies in the next decade, a theoretical framework for reasoning about modern neural networks is needed. While efforts are increasing toward demystifying why deep learning is so effective, a comprehensive picture remains lacking, suggesting that a better theory is possible. We argue that a future deep learning theory should inherit three characteristics: a \textit{hierarchically} structured network architecture, parameters \textit{iteratively} optimized using stochastic gradient-based methods, and information from the data that evolves \textit{compressively}. As an instantiation, we integrate these characteristics into a graphical model called \textit{neurashed}. This model effectively explains some common empirical patterns in deep learning. In particular, neurashed enables insights into implicit regularization, information bottleneck, and local elasticity. Finally, we discuss how neurashed can guide the development of deep learning theories. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2112_09741 |
| institution | arXiv |
| publishDate | 2021 |
| record_format | arxiv |
| spellingShingle | Envisioning Future Deep Learning Theories: Some Basic Concepts and Characteristics Su, Weijie J. Machine Learning Disordered Systems and Neural Networks Statistical Mechanics Computer Vision and Pattern Recognition To advance deep learning methodologies in the next decade, a theoretical framework for reasoning about modern neural networks is needed. While efforts are increasing toward demystifying why deep learning is so effective, a comprehensive picture remains lacking, suggesting that a better theory is possible. We argue that a future deep learning theory should inherit three characteristics: a \textit{hierarchically} structured network architecture, parameters \textit{iteratively} optimized using stochastic gradient-based methods, and information from the data that evolves \textit{compressively}. As an instantiation, we integrate these characteristics into a graphical model called \textit{neurashed}. This model effectively explains some common empirical patterns in deep learning. In particular, neurashed enables insights into implicit regularization, information bottleneck, and local elasticity. Finally, we discuss how neurashed can guide the development of deep learning theories. |
| title | Envisioning Future Deep Learning Theories: Some Basic Concepts and Characteristics |
| topic | Machine Learning Disordered Systems and Neural Networks Statistical Mechanics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2112.09741 |