Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.12491 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914095640346624 |
|---|---|
| author | Zeng, Hansi Hui, Kai Zhuang, Honglei Qin, Zhen Yue, Zhenrui Zamani, Hamed Alon, Dana |
| author_facet | Zeng, Hansi Hui, Kai Zhuang, Honglei Qin, Zhen Yue, Zhenrui Zamani, Hamed Alon, Dana |
| contents | While metrics available during pre-training, such as perplexity, correlate well with model performance at scaling-laws studies, their predictive capacities at a fixed model size remain unclear, hindering effective model selection and development. To address this gap, we formulate the task of selecting pre-training checkpoints to maximize downstream fine-tuning performance as a pairwise classification problem: predicting which of two LLMs, differing in their pre-training, will perform better after supervised fine-tuning (SFT). We construct a dataset using 50 1B parameter LLM variants with systematically varied pre-training configurations, e.g., objectives or data, and evaluate them on diverse downstream tasks after SFT. We first conduct a study and demonstrate that the conventional perplexity is a misleading indicator. As such, we introduce novel unsupervised and supervised proxy metrics derived from pre-training that successfully reduce the relative performance prediction error rate by over 50%. Despite the inherent complexity of this task, we demonstrate the practical utility of our proposed proxies in specific scenarios, paving the way for more efficient design of pre-training schemes optimized for various downstream tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_12491 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Can Pre-training Indicators Reliably Predict Fine-tuning Outcomes of LLMs? Zeng, Hansi Hui, Kai Zhuang, Honglei Qin, Zhen Yue, Zhenrui Zamani, Hamed Alon, Dana Computation and Language While metrics available during pre-training, such as perplexity, correlate well with model performance at scaling-laws studies, their predictive capacities at a fixed model size remain unclear, hindering effective model selection and development. To address this gap, we formulate the task of selecting pre-training checkpoints to maximize downstream fine-tuning performance as a pairwise classification problem: predicting which of two LLMs, differing in their pre-training, will perform better after supervised fine-tuning (SFT). We construct a dataset using 50 1B parameter LLM variants with systematically varied pre-training configurations, e.g., objectives or data, and evaluate them on diverse downstream tasks after SFT. We first conduct a study and demonstrate that the conventional perplexity is a misleading indicator. As such, we introduce novel unsupervised and supervised proxy metrics derived from pre-training that successfully reduce the relative performance prediction error rate by over 50%. Despite the inherent complexity of this task, we demonstrate the practical utility of our proposed proxies in specific scenarios, paving the way for more efficient design of pre-training schemes optimized for various downstream tasks. |
| title | Can Pre-training Indicators Reliably Predict Fine-tuning Outcomes of LLMs? |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.12491 |