A Scaling Law for Token Efficiency in LLM Fine-Tuning Under Fixed Compute Budgets
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913871802925056 |
|---|---|
| author | Lagasse, Ryan Kierans, Aidan Ghosh, Avijit Dori-Hacohen, Shiri |
| author_facet | Lagasse, Ryan Kierans, Aidan Ghosh, Avijit Dori-Hacohen, Shiri |
| contents | We introduce a scaling law for fine-tuning large language models (LLMs) under fixed compute budgets that explicitly accounts for data composition. Conventional approaches measure training data solely by total tokens, yet the number of examples and their average token length -- what we term \emph{dataset volume} -- play a decisive role in model performance. Our formulation is tuned following established procedures. Experiments on the BRICC dataset \cite{salavati2024reducing} and subsets of the MMLU dataset \cite{hendrycks2021measuringmassivemultitasklanguage}, evaluated under multiple subsampling strategies, reveal that data composition significantly affects token efficiency. These results motivate refined scaling laws for practical LLM fine-tuning in resource-constrained settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_06150 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Scaling Law for Token Efficiency in LLM Fine-Tuning Under Fixed Compute Budgets Lagasse, Ryan Kierans, Aidan Ghosh, Avijit Dori-Hacohen, Shiri Computation and Language Artificial Intelligence We introduce a scaling law for fine-tuning large language models (LLMs) under fixed compute budgets that explicitly accounts for data composition. Conventional approaches measure training data solely by total tokens, yet the number of examples and their average token length -- what we term \emph{dataset volume} -- play a decisive role in model performance. Our formulation is tuned following established procedures. Experiments on the BRICC dataset \cite{salavati2024reducing} and subsets of the MMLU dataset \cite{hendrycks2021measuringmassivemultitasklanguage}, evaluated under multiple subsampling strategies, reveal that data composition significantly affects token efficiency. These results motivate refined scaling laws for practical LLM fine-tuning in resource-constrained settings. |
| title | A Scaling Law for Token Efficiency in LLM Fine-Tuning Under Fixed Compute Budgets |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2505.06150 |