A Library of Mirrors: Deep Neural Nets in Low Dimensions are Convex Lasso Models with Reflection Features
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910539492360192 |
|---|---|
| author | Zeger, Emi Wang, Yifei Mishkin, Aaron Ergen, Tolga Candès, Emmanuel Pilanci, Mert |
| author_facet | Zeger, Emi Wang, Yifei Mishkin, Aaron Ergen, Tolga Candès, Emmanuel Pilanci, Mert |
| contents | We prove that training neural networks on 1-D data is equivalent to solving convex Lasso problems with discrete, explicitly defined dictionary matrices. We consider neural networks with piecewise linear activations and depths ranging from 2 to an arbitrary but finite number of layers. We first show that two-layer networks with piecewise linear activations are equivalent to Lasso models using a discrete dictionary of ramp functions, with breakpoints corresponding to the training data points. In certain general architectures with absolute value or ReLU activations, a third layer surprisingly creates features that reflect the training data about themselves. Additional layers progressively generate reflections of these reflections. The Lasso representation provides valuable insights into the analysis of globally optimal networks, elucidating their solution landscapes and enabling closed-form solutions in certain special cases. Numerical results show that reflections also occur when optimizing standard deep networks using standard non-convex optimizers. Additionally, we demonstrate our theory with autoregressive time series models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_01046 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Library of Mirrors: Deep Neural Nets in Low Dimensions are Convex Lasso Models with Reflection Features Zeger, Emi Wang, Yifei Mishkin, Aaron Ergen, Tolga Candès, Emmanuel Pilanci, Mert Machine Learning Artificial Intelligence Neural and Evolutionary Computing Optimization and Control We prove that training neural networks on 1-D data is equivalent to solving convex Lasso problems with discrete, explicitly defined dictionary matrices. We consider neural networks with piecewise linear activations and depths ranging from 2 to an arbitrary but finite number of layers. We first show that two-layer networks with piecewise linear activations are equivalent to Lasso models using a discrete dictionary of ramp functions, with breakpoints corresponding to the training data points. In certain general architectures with absolute value or ReLU activations, a third layer surprisingly creates features that reflect the training data about themselves. Additional layers progressively generate reflections of these reflections. The Lasso representation provides valuable insights into the analysis of globally optimal networks, elucidating their solution landscapes and enabling closed-form solutions in certain special cases. Numerical results show that reflections also occur when optimizing standard deep networks using standard non-convex optimizers. Additionally, we demonstrate our theory with autoregressive time series models. |
| title | A Library of Mirrors: Deep Neural Nets in Low Dimensions are Convex Lasso Models with Reflection Features |
| topic | Machine Learning Artificial Intelligence Neural and Evolutionary Computing Optimization and Control |
| url | https://arxiv.org/abs/2403.01046 |