Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918093429669888 |
|---|---|
| author | Li, Houyi Zheng, Wenzhen Wang, Qiufeng Ding, Zhenyu Wang, Haoying Wang, Zili Xuyang, Shijie Ding, Ning Zhou, Shuigeng Zhang, Xiangyu Jiang, Daxin |
| author_facet | Li, Houyi Zheng, Wenzhen Wang, Qiufeng Ding, Zhenyu Wang, Haoying Wang, Zili Xuyang, Shijie Ding, Ning Zhou, Shuigeng Zhang, Xiangyu Jiang, Daxin |
| contents | Training Large Language Models (LLMs) is prohibitively expensive, creating a critical scaling gap where insights from small-scale experiments often fail to transfer to resource-intensive production systems, thereby hindering efficient innovation. To bridge this, we introduce Farseer, a novel and refined scaling law offering enhanced predictive accuracy across scales. By systematically constructing a model loss surface $L(N,D)$, Farseer achieves a significantly better fit to empirical data than prior laws (e.g., Chinchilla's law). Our methodology yields accurate, robust, and highly generalizable predictions, demonstrating excellent extrapolation capabilities, improving upon Chinchilla's law by reducing extrapolation error by 433\%. This allows for the reliable evaluation of competing training strategies across all $(N,D)$ settings, enabling conclusions from small-scale ablation studies to be confidently extrapolated to predict large-scale performance. Furthermore, Farseer provides new insights into optimal compute allocation, better reflecting the nuanced demands of modern LLM training. To validate our approach, we trained an extensive suite of approximately 1,000 LLMs across diverse scales and configurations, consuming roughly 3 million NVIDIA H100 GPU hours. We are comprehensively open-sourcing all models, data, results, and logs at https://github.com/Farseer-Scaling-Law/Farseer to foster further research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_10972 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models Li, Houyi Zheng, Wenzhen Wang, Qiufeng Ding, Zhenyu Wang, Haoying Wang, Zili Xuyang, Shijie Ding, Ning Zhou, Shuigeng Zhang, Xiangyu Jiang, Daxin Machine Learning Artificial Intelligence I.2 Training Large Language Models (LLMs) is prohibitively expensive, creating a critical scaling gap where insights from small-scale experiments often fail to transfer to resource-intensive production systems, thereby hindering efficient innovation. To bridge this, we introduce Farseer, a novel and refined scaling law offering enhanced predictive accuracy across scales. By systematically constructing a model loss surface $L(N,D)$, Farseer achieves a significantly better fit to empirical data than prior laws (e.g., Chinchilla's law). Our methodology yields accurate, robust, and highly generalizable predictions, demonstrating excellent extrapolation capabilities, improving upon Chinchilla's law by reducing extrapolation error by 433\%. This allows for the reliable evaluation of competing training strategies across all $(N,D)$ settings, enabling conclusions from small-scale ablation studies to be confidently extrapolated to predict large-scale performance. Furthermore, Farseer provides new insights into optimal compute allocation, better reflecting the nuanced demands of modern LLM training. To validate our approach, we trained an extensive suite of approximately 1,000 LLMs across diverse scales and configurations, consuming roughly 3 million NVIDIA H100 GPU hours. We are comprehensively open-sourcing all models, data, results, and logs at https://github.com/Farseer-Scaling-Law/Farseer to foster further research. |
| title | Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models |
| topic | Machine Learning Artificial Intelligence I.2 |
| url | https://arxiv.org/abs/2506.10972 |