Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913123526508544 |
|---|---|
| author | Dandi, Yatin Vilucchio, Matteo Arnaboldi, Luca Tabanelli, Hugo Krzakala, Florent |
| author_facet | Dandi, Yatin Vilucchio, Matteo Arnaboldi, Luca Tabanelli, Hugo Krzakala, Florent |
| contents | Understanding how deep neural networks learn useful internal representations from data remains a central open problem in the theory of deep learning. We introduce Neural Low-Degree Filtering (Neural LoFi), a stylized limit of gradient-based training in which hierarchical feature learning becomes an explicit iterative spectral procedure. In this limit, the dynamics at each layer decouple: given the current representation, the next layer selects directions with maximal accessible low-degree correlation to the label. This yields a tractable surrogate mechanism for deep learning, together with a natural kernel-space interpretation. Neural LoFi provides a mathematically explicit framework for studying multi-layer feature learning beyond the lazy regime. It predicts how representations are selected layer by layer, explains how emergence of concepts arises with given sample complexity,and gives a concrete mechanism by which depth progressively constructs new features from old ones through low-degree compositionality. We complement the theory with mechanistic experiments on fully connected and convolutional architectures, showing that Neural LoFi improves over lazy random-feature baselines, recovers meaningful structured filters, and predicts representations aligned with early gradient-descent feature discovery with real datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_13612 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning Dandi, Yatin Vilucchio, Matteo Arnaboldi, Luca Tabanelli, Hugo Krzakala, Florent Machine Learning Disordered Systems and Neural Networks Understanding how deep neural networks learn useful internal representations from data remains a central open problem in the theory of deep learning. We introduce Neural Low-Degree Filtering (Neural LoFi), a stylized limit of gradient-based training in which hierarchical feature learning becomes an explicit iterative spectral procedure. In this limit, the dynamics at each layer decouple: given the current representation, the next layer selects directions with maximal accessible low-degree correlation to the label. This yields a tractable surrogate mechanism for deep learning, together with a natural kernel-space interpretation. Neural LoFi provides a mathematically explicit framework for studying multi-layer feature learning beyond the lazy regime. It predicts how representations are selected layer by layer, explains how emergence of concepts arises with given sample complexity,and gives a concrete mechanism by which depth progressively constructs new features from old ones through low-degree compositionality. We complement the theory with mechanistic experiments on fully connected and convolutional architectures, showing that Neural LoFi improves over lazy random-feature baselines, recovers meaningful structured filters, and predicts representations aligned with early gradient-descent feature discovery with real datasets. |
| title | Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning |
| topic | Machine Learning Disordered Systems and Neural Networks |
| url | https://arxiv.org/abs/2605.13612 |