Scaling Laws for Code: A More Data-Hungry Regime
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917503905562624 |
|---|---|
| author | Luo, Xianzhen Zheng, Wenzhen Zhu, Qingfu Zhang, Rongyi Li, Houyi Huang, Siming Fan, YuanTao Che, Wanxiang |
| author_facet | Luo, Xianzhen Zheng, Wenzhen Zhu, Qingfu Zhang, Rongyi Li, Houyi Huang, Siming Fan, YuanTao Che, Wanxiang |
| contents | Code Large Language Models (LLMs) are revolutionizing software engineering. However, scaling laws that guide the efficient training are predominantly analyzed on Natural Language (NL). Given the fundamental differences like strict syntax between code and NL, it is unclear whether these laws are directly applicable to code. To address this gap, we conduct the first large-scale empirical study of scaling laws for code, comprising 117 experimental runs with model sizes from 0.2B to 3.8B and training tokens from 2B to 128B. We fit the Chinchilla law and the Farsser law. First, the results show that the more expressive Farseer law offers greater accuracy. Second, the analysis reveals that Code LLMs scale effectively with model size. Crucially, code represents a more data-hungry regime, requiring a substantially higher data-to-parameter ratio than NL. Finally, two additional sets of experiments on code-NL mixtures show that NL benefits resource-constrained scenarios, but becomes a detriment at higher compute budgets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_08702 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scaling Laws for Code: A More Data-Hungry Regime Luo, Xianzhen Zheng, Wenzhen Zhu, Qingfu Zhang, Rongyi Li, Houyi Huang, Siming Fan, YuanTao Che, Wanxiang Computation and Language Code Large Language Models (LLMs) are revolutionizing software engineering. However, scaling laws that guide the efficient training are predominantly analyzed on Natural Language (NL). Given the fundamental differences like strict syntax between code and NL, it is unclear whether these laws are directly applicable to code. To address this gap, we conduct the first large-scale empirical study of scaling laws for code, comprising 117 experimental runs with model sizes from 0.2B to 3.8B and training tokens from 2B to 128B. We fit the Chinchilla law and the Farsser law. First, the results show that the more expressive Farseer law offers greater accuracy. Second, the analysis reveals that Code LLMs scale effectively with model size. Crucially, code represents a more data-hungry regime, requiring a substantially higher data-to-parameter ratio than NL. Finally, two additional sets of experiments on code-NL mixtures show that NL benefits resource-constrained scenarios, but becomes a detriment at higher compute budgets. |
| title | Scaling Laws for Code: A More Data-Hungry Regime |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.08702 |