D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911900646768640 |
|---|---|
| author | Que, Haoran Liu, Jiaheng Zhang, Ge Zhang, Chenchen Qu, Xingwei Ma, Yinghao Duan, Feiyu Bai, Zhiqi Wang, Jiakai Zhang, Yuanxing Tan, Xu Fu, Jie Su, Wenbo Wang, Jiamang Qu, Lin Zheng, Bo |
| author_facet | Que, Haoran Liu, Jiaheng Zhang, Ge Zhang, Chenchen Qu, Xingwei Ma, Yinghao Duan, Feiyu Bai, Zhiqi Wang, Jiakai Zhang, Yuanxing Tan, Xu Fu, Jie Su, Wenbo Wang, Jiamang Qu, Lin Zheng, Bo |
| contents | Continual Pre-Training (CPT) on Large Language Models (LLMs) has been widely used to expand the model's fundamental understanding of specific downstream domains (e.g., math and code). For the CPT on domain-specific LLMs, one important question is how to choose the optimal mixture ratio between the general-corpus (e.g., Dolma, Slim-pajama) and the downstream domain-corpus. Existing methods usually adopt laborious human efforts by grid-searching on a set of mixture ratios, which require high GPU training consumption costs. Besides, we cannot guarantee the selected ratio is optimal for the specific domain. To address the limitations of existing methods, inspired by the Scaling Law for performance prediction, we propose to investigate the Scaling Law of the Domain-specific Continual Pre-Training (D-CPT Law) to decide the optimal mixture ratio with acceptable training costs for LLMs of different sizes. Specifically, by fitting the D-CPT Law, we can easily predict the general and downstream performance of arbitrary mixture ratios, model sizes, and dataset sizes using small-scale training costs on limited experiments. Moreover, we also extend our standard D-CPT Law on cross-domain settings and propose the Cross-Domain D-CPT Law to predict the D-CPT law of target domains, where very small training costs (about 1% of the normal training costs) are needed for the target domains. Comprehensive experimental results on six downstream domains demonstrate the effectiveness and generalizability of our proposed D-CPT Law and Cross-Domain D-CPT Law. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_01375 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models Que, Haoran Liu, Jiaheng Zhang, Ge Zhang, Chenchen Qu, Xingwei Ma, Yinghao Duan, Feiyu Bai, Zhiqi Wang, Jiakai Zhang, Yuanxing Tan, Xu Fu, Jie Su, Wenbo Wang, Jiamang Qu, Lin Zheng, Bo Computation and Language Continual Pre-Training (CPT) on Large Language Models (LLMs) has been widely used to expand the model's fundamental understanding of specific downstream domains (e.g., math and code). For the CPT on domain-specific LLMs, one important question is how to choose the optimal mixture ratio between the general-corpus (e.g., Dolma, Slim-pajama) and the downstream domain-corpus. Existing methods usually adopt laborious human efforts by grid-searching on a set of mixture ratios, which require high GPU training consumption costs. Besides, we cannot guarantee the selected ratio is optimal for the specific domain. To address the limitations of existing methods, inspired by the Scaling Law for performance prediction, we propose to investigate the Scaling Law of the Domain-specific Continual Pre-Training (D-CPT Law) to decide the optimal mixture ratio with acceptable training costs for LLMs of different sizes. Specifically, by fitting the D-CPT Law, we can easily predict the general and downstream performance of arbitrary mixture ratios, model sizes, and dataset sizes using small-scale training costs on limited experiments. Moreover, we also extend our standard D-CPT Law on cross-domain settings and propose the Cross-Domain D-CPT Law to predict the D-CPT law of target domains, where very small training costs (about 1% of the normal training costs) are needed for the target domains. Comprehensive experimental results on six downstream domains demonstrate the effectiveness and generalizability of our proposed D-CPT Law and Cross-Domain D-CPT Law. |
| title | D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.01375 |