Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2304.06875 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929303887806464 |
|---|---|
| author | Yao, Yiqun fan, Siqi Huang, Xiusheng Fang, Xuezhi Li, Xiang Ni, Ziyi Jiang, Xin Meng, Xuying Han, Peng Shang, Shuo Liu, Kang Sun, Aixin Wang, Yequan |
| author_facet | Yao, Yiqun fan, Siqi Huang, Xiusheng Fang, Xuezhi Li, Xiang Ni, Ziyi Jiang, Xin Meng, Xuying Han, Peng Shang, Shuo Liu, Kang Sun, Aixin Wang, Yequan |
| contents | As language models scale up, it becomes increasingly expensive to verify research ideas because conclusions on small models do not trivially transfer to large ones. A possible solution is to establish a generic system that accurately predicts certain metrics for large models without training them. Existing scaling laws require hyperparameter search on the largest models, limiting their predicative capability. In this paper, we present an approach (namely μScaling) to predict the pre-training loss, based on our observations that Maximal Update Parametrization (μP) enables accurate fitting of scaling laws close to common loss basins in hyperparameter space. With μScaling, different model designs can be compared on large scales by training only their smaller counterparts. Further, we introduce nanoLM: an affordable LLM pre-training benchmark that facilitates this new research paradigm. With around 14% of the one-time pre-training cost, we can accurately forecast the loss for models up to 52B. Our goal with nanoLM is to empower researchers with limited resources to reach meaningful conclusions on large models. We also aspire for our benchmark to serve as a bridge between the academic community and the industry. Code for μScaling is available at https://github.com/cofe-ai/Mu-scaling. Code for nanoLLM will be available later. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2304_06875 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | nanoLM: an Affordable LLM Pre-training Benchmark via Accurate Loss Prediction across Scales Yao, Yiqun fan, Siqi Huang, Xiusheng Fang, Xuezhi Li, Xiang Ni, Ziyi Jiang, Xin Meng, Xuying Han, Peng Shang, Shuo Liu, Kang Sun, Aixin Wang, Yequan Computation and Language Machine Learning As language models scale up, it becomes increasingly expensive to verify research ideas because conclusions on small models do not trivially transfer to large ones. A possible solution is to establish a generic system that accurately predicts certain metrics for large models without training them. Existing scaling laws require hyperparameter search on the largest models, limiting their predicative capability. In this paper, we present an approach (namely μScaling) to predict the pre-training loss, based on our observations that Maximal Update Parametrization (μP) enables accurate fitting of scaling laws close to common loss basins in hyperparameter space. With μScaling, different model designs can be compared on large scales by training only their smaller counterparts. Further, we introduce nanoLM: an affordable LLM pre-training benchmark that facilitates this new research paradigm. With around 14% of the one-time pre-training cost, we can accurately forecast the loss for models up to 52B. Our goal with nanoLM is to empower researchers with limited resources to reach meaningful conclusions on large models. We also aspire for our benchmark to serve as a bridge between the academic community and the industry. Code for μScaling is available at https://github.com/cofe-ai/Mu-scaling. Code for nanoLLM will be available later. |
| title | nanoLM: an Affordable LLM Pre-training Benchmark via Accurate Loss Prediction across Scales |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2304.06875 |