Superposition unifies power-law training dynamics
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918317954957312 |
|---|---|
| author | Chen, Zixin Jessie Chen, Hao Liu, Yizhou Gore, Jeff |
| author_facet | Chen, Zixin Jessie Chen, Hao Liu, Yizhou Gore, Jeff |
| contents | We investigate the role of feature superposition in the emergence of power-law training dynamics using a teacher-student framework. We first derive an analytic theory for training without superposition, establishing that the power-law training exponent depends on both the input data statistics and channel importance. Remarkably, we discover that a superposition bottleneck induces a transition to a universal power-law exponent of $\sim 1$, independent of data and channel statistics. This one over time training with superposition represents an up to tenfold acceleration compared to the purely sequential learning that takes place in the absence of superposition. Our finding that superposition leads to rapid training with a data-independent power law exponent may have important implications for a wide range of neural networks that employ superposition, including production-scale large language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_01045 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Superposition unifies power-law training dynamics Chen, Zixin Jessie Chen, Hao Liu, Yizhou Gore, Jeff Machine Learning Artificial Intelligence Data Analysis, Statistics and Probability We investigate the role of feature superposition in the emergence of power-law training dynamics using a teacher-student framework. We first derive an analytic theory for training without superposition, establishing that the power-law training exponent depends on both the input data statistics and channel importance. Remarkably, we discover that a superposition bottleneck induces a transition to a universal power-law exponent of $\sim 1$, independent of data and channel statistics. This one over time training with superposition represents an up to tenfold acceleration compared to the purely sequential learning that takes place in the absence of superposition. Our finding that superposition leads to rapid training with a data-independent power law exponent may have important implications for a wide range of neural networks that employ superposition, including production-scale large language models. |
| title | Superposition unifies power-law training dynamics |
| topic | Machine Learning Artificial Intelligence Data Analysis, Statistics and Probability |
| url | https://arxiv.org/abs/2602.01045 |