Resolving Discrepancies in Compute-Optimal Scaling of Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915108851023872 |
|---|---|
| author | Porian, Tomer Wortsman, Mitchell Jitsev, Jenia Schmidt, Ludwig Carmon, Yair |
| author_facet | Porian, Tomer Wortsman, Mitchell Jitsev, Jenia Schmidt, Ludwig Carmon, Yair |
| contents | Kaplan et al. and Hoffmann et al. developed influential scaling laws for the optimal model size as a function of the compute budget, but these laws yield substantially different predictions. We explain the discrepancy by reproducing the Kaplan scaling law on two datasets (OpenWebText2 and RefinedWeb) and identifying three factors causing the difference: last layer computational cost, warmup duration, and scale-dependent optimizer tuning. With these factors corrected, we obtain excellent agreement with the Hoffmann et al. (i.e., "Chinchilla") scaling law. Counter to a hypothesis of Hoffmann et al., we find that careful learning rate decay is not essential for the validity of their scaling law. As a secondary result, we derive scaling laws for the optimal learning rate and batch size, finding that tuning the AdamW $β_2$ parameter is essential at lower batch sizes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_19146 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Resolving Discrepancies in Compute-Optimal Scaling of Language Models Porian, Tomer Wortsman, Mitchell Jitsev, Jenia Schmidt, Ludwig Carmon, Yair Machine Learning Computation and Language Kaplan et al. and Hoffmann et al. developed influential scaling laws for the optimal model size as a function of the compute budget, but these laws yield substantially different predictions. We explain the discrepancy by reproducing the Kaplan scaling law on two datasets (OpenWebText2 and RefinedWeb) and identifying three factors causing the difference: last layer computational cost, warmup duration, and scale-dependent optimizer tuning. With these factors corrected, we obtain excellent agreement with the Hoffmann et al. (i.e., "Chinchilla") scaling law. Counter to a hypothesis of Hoffmann et al., we find that careful learning rate decay is not essential for the validity of their scaling law. As a secondary result, we derive scaling laws for the optimal learning rate and batch size, finding that tuning the AdamW $β_2$ parameter is essential at lower batch sizes. |
| title | Resolving Discrepancies in Compute-Optimal Scaling of Language Models |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2406.19146 |