Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.00276 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912799919177728 |
|---|---|
| author | Li, Hongxi Huang, Chunlin |
| author_facet | Li, Hongxi Huang, Chunlin |
| contents | We present a theory of feature learning in wide L2-regularized networks showing that supervised learning is inherently compressive. We derive a kernel ODE that predicts a "water-filling" spectral evolution and prove that for any stable steady state, the kernel rank is bounded by the number of classes ($C$). We further demonstrate that SGD noise is similarly low-rank ($O(C)$), confining dynamics to the task-relevant subspace. This framework unifies the deterministic and stochastic views of alignment and contrasts the low-rank nature of supervised learning with the high-rank, expansive representations of self-supervision. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_00276 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Task-Driven Kernel Flows: Label Rank Compression and Laplacian Spectral Filtering Li, Hongxi Huang, Chunlin Machine Learning We present a theory of feature learning in wide L2-regularized networks showing that supervised learning is inherently compressive. We derive a kernel ODE that predicts a "water-filling" spectral evolution and prove that for any stable steady state, the kernel rank is bounded by the number of classes ($C$). We further demonstrate that SGD noise is similarly low-rank ($O(C)$), confining dynamics to the task-relevant subspace. This framework unifies the deterministic and stochastic views of alignment and contrasts the low-rank nature of supervised learning with the high-rank, expansive representations of self-supervision. |
| title | Task-Driven Kernel Flows: Label Rank Compression and Laplacian Spectral Filtering |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2601.00276 |