Gemstones: A Model Suite for Multi-Faceted Scaling Laws
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909829397741568 |
|---|---|
| author | McLeish, Sean Kirchenbauer, John Miller, David Yu Singh, Siddharth Bhatele, Abhinav Goldblum, Micah Panda, Ashwinee Goldstein, Tom |
| author_facet | McLeish, Sean Kirchenbauer, John Miller, David Yu Singh, Siddharth Bhatele, Abhinav Goldblum, Micah Panda, Ashwinee Goldstein, Tom |
| contents | Scaling laws are typically fit using a family of models with a narrow range of frozen hyperparameter choices. In this work we study scaling laws using multiple architectural shapes and hyperparameter choices, highlighting their impact on resulting prescriptions. As a primary artifact of our research, we release the Gemstones: an open-source scaling law dataset, consisting of over 4000 checkpoints from transformers with up to 2 billion parameters and diverse architectural shapes; including ablations over learning rate and cooldown. Our checkpoints enable more complex studies of scaling, such as analyzing the relationship between width and depth. By examining our model suite, we find that the prescriptions of scaling laws can be highly sensitive to the experimental design process and the specific model checkpoints used during fitting. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_06857 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Gemstones: A Model Suite for Multi-Faceted Scaling Laws McLeish, Sean Kirchenbauer, John Miller, David Yu Singh, Siddharth Bhatele, Abhinav Goldblum, Micah Panda, Ashwinee Goldstein, Tom Machine Learning Artificial Intelligence Scaling laws are typically fit using a family of models with a narrow range of frozen hyperparameter choices. In this work we study scaling laws using multiple architectural shapes and hyperparameter choices, highlighting their impact on resulting prescriptions. As a primary artifact of our research, we release the Gemstones: an open-source scaling law dataset, consisting of over 4000 checkpoints from transformers with up to 2 billion parameters and diverse architectural shapes; including ablations over learning rate and cooldown. Our checkpoints enable more complex studies of scaling, such as analyzing the relationship between width and depth. By examining our model suite, we find that the prescriptions of scaling laws can be highly sensitive to the experimental design process and the specific model checkpoints used during fitting. |
| title | Gemstones: A Model Suite for Multi-Faceted Scaling Laws |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2502.06857 |