Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908449324924928 |
|---|---|
| author | Mircea, Andrei Chakraborty, Supriyo Chitsazan, Nima Naphade, Milind Sahu, Sambit Rish, Irina Lobacheva, Ekaterina |
| author_facet | Mircea, Andrei Chakraborty, Supriyo Chitsazan, Nima Naphade, Milind Sahu, Sambit Rish, Irina Lobacheva, Ekaterina |
| contents | This work aims to understand how scaling improves language models, specifically in terms of training dynamics. We find that language models undergo loss deceleration early in training; an abrupt slowdown in the rate of loss improvement, resulting in piecewise linear behaviour of the loss curve in log-log space. Scaling up the model mitigates this transition by (1) decreasing the loss at which deceleration occurs, and (2) improving the log-log rate of loss improvement after deceleration. We attribute loss deceleration to a type of degenerate training dynamics we term zero-sum learning (ZSL). In ZSL, per-example gradients become systematically opposed, leading to destructive interference in per-example changes in loss. As a result, improving loss on one subset of examples degrades it on another, bottlenecking overall progress. Loss deceleration and ZSL provide new insights into the training dynamics underlying language model scaling laws, and could potentially be targeted directly to improve language models independent of scale. We make our code and artefacts available at: https://github.com/mirandrom/zsl |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_05447 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learning Mircea, Andrei Chakraborty, Supriyo Chitsazan, Nima Naphade, Milind Sahu, Sambit Rish, Irina Lobacheva, Ekaterina Machine Learning Artificial Intelligence I.2.7 This work aims to understand how scaling improves language models, specifically in terms of training dynamics. We find that language models undergo loss deceleration early in training; an abrupt slowdown in the rate of loss improvement, resulting in piecewise linear behaviour of the loss curve in log-log space. Scaling up the model mitigates this transition by (1) decreasing the loss at which deceleration occurs, and (2) improving the log-log rate of loss improvement after deceleration. We attribute loss deceleration to a type of degenerate training dynamics we term zero-sum learning (ZSL). In ZSL, per-example gradients become systematically opposed, leading to destructive interference in per-example changes in loss. As a result, improving loss on one subset of examples degrades it on another, bottlenecking overall progress. Loss deceleration and ZSL provide new insights into the training dynamics underlying language model scaling laws, and could potentially be targeted directly to improve language models independent of scale. We make our code and artefacts available at: https://github.com/mirandrom/zsl |
| title | Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learning |
| topic | Machine Learning Artificial Intelligence I.2.7 |
| url | https://arxiv.org/abs/2506.05447 |