Can Language Models Discover Scaling Laws?
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912839459930112 |
|---|---|
| author | Lin, Haowei Ye, Haotian Feng, Wenzheng Huang, Quzhe Li, Yujun Lim, Hubert Li, Zhengrui Wang, Xiangyu Ma, Jianzhu Liang, Yitao Zou, James |
| author_facet | Lin, Haowei Ye, Haotian Feng, Wenzheng Huang, Quzhe Li, Yujun Lim, Hubert Li, Zhengrui Wang, Xiangyu Ma, Jianzhu Liang, Yitao Zou, James |
| contents | Discovering scaling laws for predicting model performance at scale is a fundamental and open-ended challenge, mostly reliant on slow, case specific human experimentation. To investigate the potential for LLMs to automate this process, we collect over 5,000 experiments from existing literature and curate eight diverse scaling law discovery tasks. While existing agents struggle to produce accurate law formulas, this paper introduces SLDAgent, an evolution-based agent that co-optimize the scaling law model and the parameters, enabling it to autonomously explore complex relationships between variables. For the first time, we demonstrates that SLDAgent can automatically discover laws that exhibit consistently more accurate extrapolation than their established, human-derived counterparts across all tasks. Through comprehensive analysis, we elucidate why these discovered laws are superior and verify their practical utility in both pretraining and finetuning applications. This work establishes a new paradigm for agentic scientific discovery, showing that AI systems can understand their own scaling behavior, and can contribute novel and practical knowledge back to the research community. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_21184 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Can Language Models Discover Scaling Laws? Lin, Haowei Ye, Haotian Feng, Wenzheng Huang, Quzhe Li, Yujun Lim, Hubert Li, Zhengrui Wang, Xiangyu Ma, Jianzhu Liang, Yitao Zou, James Machine Learning Artificial Intelligence Computation and Language Discovering scaling laws for predicting model performance at scale is a fundamental and open-ended challenge, mostly reliant on slow, case specific human experimentation. To investigate the potential for LLMs to automate this process, we collect over 5,000 experiments from existing literature and curate eight diverse scaling law discovery tasks. While existing agents struggle to produce accurate law formulas, this paper introduces SLDAgent, an evolution-based agent that co-optimize the scaling law model and the parameters, enabling it to autonomously explore complex relationships between variables. For the first time, we demonstrates that SLDAgent can automatically discover laws that exhibit consistently more accurate extrapolation than their established, human-derived counterparts across all tasks. Through comprehensive analysis, we elucidate why these discovered laws are superior and verify their practical utility in both pretraining and finetuning applications. This work establishes a new paradigm for agentic scientific discovery, showing that AI systems can understand their own scaling behavior, and can contribute novel and practical knowledge back to the research community. |
| title | Can Language Models Discover Scaling Laws? |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2507.21184 |