Influence Dynamics and Stagewise Data Attribution
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918160135880704 |
|---|---|
| author | Lee, Jin Hwa Smith, Matthew Adam, Maxwell Hoogland, Jesse |
| author_facet | Lee, Jin Hwa Smith, Matthew Adam, Maxwell Hoogland, Jesse |
| contents | Current training data attribution (TDA) methods treat the influence one sample has on another as static, but neural networks learn in distinct stages that exhibit changing patterns of influence. In this work, we introduce a framework for stagewise data attribution grounded in singular learning theory. We predict that influence can change non-monotonically, including sign flips and sharp peaks at developmental transitions. We first validate these predictions analytically and empirically in a toy model, showing that dynamic shifts in influence directly map to the model's progressive learning of a semantic hierarchy. Finally, we demonstrate these phenomena at scale in language models, where token-level influence changes align with known developmental stages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_12071 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Influence Dynamics and Stagewise Data Attribution Lee, Jin Hwa Smith, Matthew Adam, Maxwell Hoogland, Jesse Machine Learning Current training data attribution (TDA) methods treat the influence one sample has on another as static, but neural networks learn in distinct stages that exhibit changing patterns of influence. In this work, we introduce a framework for stagewise data attribution grounded in singular learning theory. We predict that influence can change non-monotonically, including sign flips and sharp peaks at developmental transitions. We first validate these predictions analytically and empirically in a toy model, showing that dynamic shifts in influence directly map to the model's progressive learning of a semantic hierarchy. Finally, we demonstrate these phenomena at scale in language models, where token-level influence changes align with known developmental stages. |
| title | Influence Dynamics and Stagewise Data Attribution |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2510.12071 |