Self-Improving Pretraining: using post-trained models to pretrain better models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910104398331904 |
|---|---|
| author | Tan, Ellen Xiaoqing Lanchantin, Jack Dhuliawala, Shehzaad Li, Danwei Nguyen, Thao Xu, Jing Yu, Ping Kulikov, Ilia Sukhbaatar, Sainbayar Weston, Jason Li, Xian Golovneva, Olga |
| author_facet | Tan, Ellen Xiaoqing Lanchantin, Jack Dhuliawala, Shehzaad Li, Danwei Nguyen, Thao Xu, Jing Yu, Ping Kulikov, Ilia Sukhbaatar, Sainbayar Weston, Jason Li, Xian Golovneva, Olga |
| contents | Large language models are classically trained in stages: pretraining on raw text followed by post-training for instruction following and reasoning. However, this separation creates a fundamental limitation: many desirable behaviors such as safety, factuality, overall generation quality, and reasoning ability are only added at a late stage, even though the patterns learned earlier strongly shape a model's capabilities. To tackle this issue, we introduce a new way to pretrain and mid-train models that incorporates these behaviors earlier. We utilize an existing strong, post-trained model to both rewrite pretraining data and to judge policy model rollouts, thus using reinforcement earlier in training. In our experiments, we show this can give strong gains in quality, safety, factuality and reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_21343 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Self-Improving Pretraining: using post-trained models to pretrain better models Tan, Ellen Xiaoqing Lanchantin, Jack Dhuliawala, Shehzaad Li, Danwei Nguyen, Thao Xu, Jing Yu, Ping Kulikov, Ilia Sukhbaatar, Sainbayar Weston, Jason Li, Xian Golovneva, Olga Computation and Language Artificial Intelligence Machine Learning Large language models are classically trained in stages: pretraining on raw text followed by post-training for instruction following and reasoning. However, this separation creates a fundamental limitation: many desirable behaviors such as safety, factuality, overall generation quality, and reasoning ability are only added at a late stage, even though the patterns learned earlier strongly shape a model's capabilities. To tackle this issue, we introduce a new way to pretrain and mid-train models that incorporates these behaviors earlier. We utilize an existing strong, post-trained model to both rewrite pretraining data and to judge policy model rollouts, thus using reinforcement earlier in training. In our experiments, we show this can give strong gains in quality, safety, factuality and reasoning. |
| title | Self-Improving Pretraining: using post-trained models to pretrain better models |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2601.21343 |