Scalable Complexity Control Facilitates Reasoning Ability of LLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915311795568640 |
|---|---|
| author | Hang, Liangkai Yao, Junjie Bai, Zhiwei Chen, Tianyi Chen, Yang Diao, Rongjie Li, Hezhou Lin, Pengxiao Wang, Zhiwei Xu, Cheng Zhang, Zhongwang Zhou, Zhangchen Li, Zhiyu Lin, Zehao Chen, Kai Xiong, Feiyu Zhang, Yaoyu E, Weinan Yang, Hongkang Xu, Zhi-Qin John |
| author_facet | Hang, Liangkai Yao, Junjie Bai, Zhiwei Chen, Tianyi Chen, Yang Diao, Rongjie Li, Hezhou Lin, Pengxiao Wang, Zhiwei Xu, Cheng Zhang, Zhongwang Zhou, Zhangchen Li, Zhiyu Lin, Zehao Chen, Kai Xiong, Feiyu Zhang, Yaoyu E, Weinan Yang, Hongkang Xu, Zhi-Qin John |
| contents | The reasoning ability of large language models (LLMs) has been rapidly advancing in recent years, attracting interest in more fundamental approaches that can reliably enhance their generalizability. This work demonstrates that model complexity control, conveniently implementable by adjusting the initialization rate and weight decay coefficient, improves the scaling law of LLMs consistently over varying model sizes and data sizes. This gain is further illustrated by comparing the benchmark performance of 2.4B models pretrained on 1T tokens with different complexity hyperparameters. Instead of fixing the initialization std, we found that a constant initialization rate (the exponent of std) enables the scaling law to descend faster in both model and data sizes. These results indicate that complexity control is a promising direction for the continual advancement of LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_23013 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scalable Complexity Control Facilitates Reasoning Ability of LLMs Hang, Liangkai Yao, Junjie Bai, Zhiwei Chen, Tianyi Chen, Yang Diao, Rongjie Li, Hezhou Lin, Pengxiao Wang, Zhiwei Xu, Cheng Zhang, Zhongwang Zhou, Zhangchen Li, Zhiyu Lin, Zehao Chen, Kai Xiong, Feiyu Zhang, Yaoyu E, Weinan Yang, Hongkang Xu, Zhi-Qin John Machine Learning The reasoning ability of large language models (LLMs) has been rapidly advancing in recent years, attracting interest in more fundamental approaches that can reliably enhance their generalizability. This work demonstrates that model complexity control, conveniently implementable by adjusting the initialization rate and weight decay coefficient, improves the scaling law of LLMs consistently over varying model sizes and data sizes. This gain is further illustrated by comparing the benchmark performance of 2.4B models pretrained on 1T tokens with different complexity hyperparameters. Instead of fixing the initialization std, we found that a constant initialization rate (the exponent of std) enables the scaling law to descend faster in both model and data sizes. These results indicate that complexity control is a promising direction for the continual advancement of LLMs. |
| title | Scalable Complexity Control Facilitates Reasoning Ability of LLMs |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2505.23013 |