On Predicting the Post-training Potential of Pre-trained LLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911674108215296 |
|---|---|
| author | Li, Xiaoyuan Ma, Yubo Yang, Kexin Li, Moxin Bao, Keqin Wang, Wenie Feng, Fuli Liu, Dayiheng |
| author_facet | Li, Xiaoyuan Ma, Yubo Yang, Kexin Li, Moxin Bao, Keqin Wang, Wenie Feng, Fuli Liu, Dayiheng |
| contents | The performance of Large Language Models (LLMs) on downstream tasks is fundamentally constrained by the capabilities acquired during pre-training. However, traditional benchmarks like MMLU often fail to reflect a base model's plasticity in complex open-ended scenarios, leading to inefficient model selection. We address this by introducing a new task of predicting post-training potential - forecasting a base model's performance before post-training. We propose RuDE (Rubric-based Discriminative Evaluation), a unified framework that bypasses the generation gap of base models by leveraging response discrimination. Guided by our systematic 4C Taxonomy, RuDE constructs controlled contrastive pairs across diverse domains by fine-grained rubric violations. Extensive experiments demonstrate a correlation greater than 90% with post-training performance. Crucially, validation via Reinforcement Learning (RL) confirms that RuDE effectively identifies high-potential smaller models that outperform larger counterparts, offering a compute-efficient mechanism for foundation model development. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_11978 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | On Predicting the Post-training Potential of Pre-trained LLMs Li, Xiaoyuan Ma, Yubo Yang, Kexin Li, Moxin Bao, Keqin Wang, Wenie Feng, Fuli Liu, Dayiheng Computation and Language The performance of Large Language Models (LLMs) on downstream tasks is fundamentally constrained by the capabilities acquired during pre-training. However, traditional benchmarks like MMLU often fail to reflect a base model's plasticity in complex open-ended scenarios, leading to inefficient model selection. We address this by introducing a new task of predicting post-training potential - forecasting a base model's performance before post-training. We propose RuDE (Rubric-based Discriminative Evaluation), a unified framework that bypasses the generation gap of base models by leveraging response discrimination. Guided by our systematic 4C Taxonomy, RuDE constructs controlled contrastive pairs across diverse domains by fine-grained rubric violations. Extensive experiments demonstrate a correlation greater than 90% with post-training performance. Crucially, validation via Reinforcement Learning (RL) confirms that RuDE effectively identifies high-potential smaller models that outperform larger counterparts, offering a compute-efficient mechanism for foundation model development. |
| title | On Predicting the Post-training Potential of Pre-trained LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.11978 |