Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.31244 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913172755054592 |
|---|---|
| author | Nikolaou, Konstantin Scheunemann, Jonas Krippendorf, Sven Tovey, Samuel Holm, Christian |
| author_facet | Nikolaou, Konstantin Scheunemann, Jonas Krippendorf, Sven Tovey, Samuel Holm, Christian |
| contents | Neural scaling laws describe predictable power-law relationships between model size, dataset size, compute, and performance. While these laws guide the development of modern foundation models, the mechanisms underpinning them remain poorly understood, in part due to the absence of scalable analysis tools. To close this gap, we introduce "spectral position": a scalable measure of which eigenvalues of the empirical neural tangent kernel (eNTK) currently drive loss reduction. Applying this measure to scaling experiments, we find that spectral position decreases throughout training: learning shifts from dominant eigenmodes into the spectral tail. Larger models reach further into the tail than smaller models, revealing a size-dependent capacity we call "spectral reach". This suggests why larger models achieve lower losses: they sustain learning on weak spectral signals inaccessible to smaller models. We further identify feature learning as a key enabler of spectral reach. It adaptively amplifies gradient magnitudes as learning advances, sustaining progress where frozen representations stall. This points to concrete interventions through architecture and optimizer design. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_31244 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tail Nikolaou, Konstantin Scheunemann, Jonas Krippendorf, Sven Tovey, Samuel Holm, Christian Machine Learning Computational Physics Neural scaling laws describe predictable power-law relationships between model size, dataset size, compute, and performance. While these laws guide the development of modern foundation models, the mechanisms underpinning them remain poorly understood, in part due to the absence of scalable analysis tools. To close this gap, we introduce "spectral position": a scalable measure of which eigenvalues of the empirical neural tangent kernel (eNTK) currently drive loss reduction. Applying this measure to scaling experiments, we find that spectral position decreases throughout training: learning shifts from dominant eigenmodes into the spectral tail. Larger models reach further into the tail than smaller models, revealing a size-dependent capacity we call "spectral reach". This suggests why larger models achieve lower losses: they sustain learning on weak spectral signals inaccessible to smaller models. We further identify feature learning as a key enabler of spectral reach. It adaptively amplifies gradient magnitudes as learning advances, sustaining progress where frozen representations stall. This points to concrete interventions through architecture and optimizer design. |
| title | Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tail |
| topic | Machine Learning Computational Physics |
| url | https://arxiv.org/abs/2605.31244 |