Establishing Task Scaling Laws via Compute-Efficient Model Ladders
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911115346182144 |
|---|---|
| author | Bhagia, Akshita Liu, Jiacheng Wettig, Alexander Heineman, David Tafjord, Oyvind Jha, Ananya Harsh Soldaini, Luca Smith, Noah A. Groeneveld, Dirk Koh, Pang Wei Dodge, Jesse Hajishirzi, Hannaneh |
| author_facet | Bhagia, Akshita Liu, Jiacheng Wettig, Alexander Heineman, David Tafjord, Oyvind Jha, Ananya Harsh Soldaini, Luca Smith, Noah A. Groeneveld, Dirk Koh, Pang Wei Dodge, Jesse Hajishirzi, Hannaneh |
| contents | We develop task scaling laws and model ladders to predict the individual task performance of pretrained language models (LMs) in the overtrained setting. Standard power laws for language modeling loss cannot accurately model task performance. Therefore, we leverage a two-step prediction approach: (1) use model and data size to predict an intermediate loss, then (2) use it to predict task performance. We train a set of small-scale "ladder" models, collect data points to fit the parameterized functions of the two prediction steps, and make predictions for two target models: a 7B model trained to 4T tokens and a 13B model trained to 5T tokens. Training the ladder models only costs 1% of the compute used for the target models. On four multiple-choice tasks formatted as ranked classification, we can predict the accuracy of both target models within 2 points of absolute error. We find that tasks with higher prediction error also have higher variance in the metrics over model checkpoints. We also contrast multiple design choices for predicting accuracy, and present recommendations for extending our method to new models and tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_04403 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Establishing Task Scaling Laws via Compute-Efficient Model Ladders Bhagia, Akshita Liu, Jiacheng Wettig, Alexander Heineman, David Tafjord, Oyvind Jha, Ananya Harsh Soldaini, Luca Smith, Noah A. Groeneveld, Dirk Koh, Pang Wei Dodge, Jesse Hajishirzi, Hannaneh Computation and Language Artificial Intelligence We develop task scaling laws and model ladders to predict the individual task performance of pretrained language models (LMs) in the overtrained setting. Standard power laws for language modeling loss cannot accurately model task performance. Therefore, we leverage a two-step prediction approach: (1) use model and data size to predict an intermediate loss, then (2) use it to predict task performance. We train a set of small-scale "ladder" models, collect data points to fit the parameterized functions of the two prediction steps, and make predictions for two target models: a 7B model trained to 4T tokens and a 13B model trained to 5T tokens. Training the ladder models only costs 1% of the compute used for the target models. On four multiple-choice tasks formatted as ranked classification, we can predict the accuracy of both target models within 2 points of absolute error. We find that tasks with higher prediction error also have higher variance in the metrics over model checkpoints. We also contrast multiple design choices for predicting accuracy, and present recommendations for extending our method to new models and tasks. |
| title | Establishing Task Scaling Laws via Compute-Efficient Model Ladders |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2412.04403 |