Learning curves theory for hierarchically compositional data with power-law distributed features
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916732134752256 |
|---|---|
| author | Cagnetta, Francesco Kang, Hyunmo Wyart, Matthieu |
| author_facet | Cagnetta, Francesco Kang, Hyunmo Wyart, Matthieu |
| contents | Recent theories suggest that Neural Scaling Laws arise whenever the task is linearly decomposed into power-law distributed units. Alternatively, scaling laws also emerge when data exhibit a hierarchically compositional structure, as is thought to occur in language and images. To unify these views, we consider classification and next-token prediction tasks based on probabilistic context-free grammars -- probabilistic models that generate data via a hierarchy of production rules. For classification, we show that having power-law distributed production rules results in a power-law learning curve with an exponent depending on the rules' distribution and a large multiplicative constant that depends on the hierarchical structure. By contrast, for next-token prediction, the distribution of production rules controls the local details of the learning curve, but not the exponent describing the large-scale behaviour. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_07067 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Learning curves theory for hierarchically compositional data with power-law distributed features Cagnetta, Francesco Kang, Hyunmo Wyart, Matthieu Machine Learning Disordered Systems and Neural Networks Recent theories suggest that Neural Scaling Laws arise whenever the task is linearly decomposed into power-law distributed units. Alternatively, scaling laws also emerge when data exhibit a hierarchically compositional structure, as is thought to occur in language and images. To unify these views, we consider classification and next-token prediction tasks based on probabilistic context-free grammars -- probabilistic models that generate data via a hierarchy of production rules. For classification, we show that having power-law distributed production rules results in a power-law learning curve with an exponent depending on the rules' distribution and a large multiplicative constant that depends on the hierarchical structure. By contrast, for next-token prediction, the distribution of production rules controls the local details of the learning curve, but not the exponent describing the large-scale behaviour. |
| title | Learning curves theory for hierarchically compositional data with power-law distributed features |
| topic | Machine Learning Disordered Systems and Neural Networks |
| url | https://arxiv.org/abs/2505.07067 |