MaP: A Unified Framework for Reliable Evaluation of Pre-training Dynamics
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914392184979456 |
|---|---|
| author | Wang, Jiapeng Tian, Changxin Chen, Kunlong Liu, Ziqi Mao, Jiaxin Zhao, Wayne Xin Zhang, Zhiqiang Zhou, Jun |
| author_facet | Wang, Jiapeng Tian, Changxin Chen, Kunlong Liu, Ziqi Mao, Jiaxin Zhao, Wayne Xin Zhang, Zhiqiang Zhou, Jun |
| contents | Reliable evaluation is fundamental to the progress of Large Language Models (LLMs), yet the evaluation process during pre-training is plagued by significant instability that obscures true learning dynamics. In this work, we systematically diagnose this instability, attributing it to two distinct sources: \textit{Parameter Instability} from training stochasticity and \textit{Evaluation Instability} from noisy measurement protocols. To counteract both sources of noise, we introduce \textbf{MaP}, a dual-pronged framework that synergistically integrates checkpoint \underline{M}erging \underline{a}nd the \underline{P}ass@k metric. Checkpoint merging smooths the parameter space by averaging recent model weights, while Pass@k provides a robust, low-variance statistical estimate of model capability. Extensive experiments show that MaP yields significantly smoother performance curves, reduces inter-run variance, and ensures more consistent model rankings. Ultimately, MaP provides a more reliable and faithful lens for observing LLM training dynamics, laying a crucial empirical foundation for LLM research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_09295 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MaP: A Unified Framework for Reliable Evaluation of Pre-training Dynamics Wang, Jiapeng Tian, Changxin Chen, Kunlong Liu, Ziqi Mao, Jiaxin Zhao, Wayne Xin Zhang, Zhiqiang Zhou, Jun Computation and Language Reliable evaluation is fundamental to the progress of Large Language Models (LLMs), yet the evaluation process during pre-training is plagued by significant instability that obscures true learning dynamics. In this work, we systematically diagnose this instability, attributing it to two distinct sources: \textit{Parameter Instability} from training stochasticity and \textit{Evaluation Instability} from noisy measurement protocols. To counteract both sources of noise, we introduce \textbf{MaP}, a dual-pronged framework that synergistically integrates checkpoint \underline{M}erging \underline{a}nd the \underline{P}ass@k metric. Checkpoint merging smooths the parameter space by averaging recent model weights, while Pass@k provides a robust, low-variance statistical estimate of model capability. Extensive experiments show that MaP yields significantly smoother performance curves, reduces inter-run variance, and ensures more consistent model rankings. Ultimately, MaP provides a more reliable and faithful lens for observing LLM training dynamics, laying a crucial empirical foundation for LLM research. |
| title | MaP: A Unified Framework for Reliable Evaluation of Pre-training Dynamics |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.09295 |