Introspective X Training: Feedback Conditioning Improves Scaling Across all LLM Training Stages
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910237199433728 |
|---|---|
| author | Cui, Brandon Lu, Ximing Jung, Jaehun Akter, Syeda Nahida Kim, Hyunwoo Qu, Yuxiao Acuna, David Prabhumoye, Shrimai Choi, Yejin Ammanabrolu, Prithviraj |
| author_facet | Cui, Brandon Lu, Ximing Jung, Jaehun Akter, Syeda Nahida Kim, Hyunwoo Qu, Yuxiao Acuna, David Prabhumoye, Shrimai Choi, Yejin Ammanabrolu, Prithviraj |
| contents | We tackle the question of how to scale more efficiently across the many, ever-growing stages of current LLM training pipelines. Our guiding intuition stems from the fact that the dynamics of later stages of the pipeline, e.g. post-training, can be used to inform earlier stages such as pre-training. To this end, we propose Introspective Training (or IXT), inspired by offline reward-conditioned reinforcement learning and applicable to any stage of training. IXT uses a thinking reward model to annotate data with natural language critique based feedback, enabling quality aware training from the earliest stages of the pipeline. Models are then trained by prefix-conditioning the data with the generated feedback -- ensuring that not all tokens are treated equally starting much earlier in training than usual. Comprehensive experiments on 7.5-12B transformer-based dense LLMs trained from scratch all the way up to 18 Trillion tokens seen show that our method: bends scaling curves resulting in up to 2.8x more compute efficiency generally; and reaches performance levels unachievable for models trained otherwise in domains such as math and code. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_20285 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Introspective X Training: Feedback Conditioning Improves Scaling Across all LLM Training Stages Cui, Brandon Lu, Ximing Jung, Jaehun Akter, Syeda Nahida Kim, Hyunwoo Qu, Yuxiao Acuna, David Prabhumoye, Shrimai Choi, Yejin Ammanabrolu, Prithviraj Machine Learning Artificial Intelligence We tackle the question of how to scale more efficiently across the many, ever-growing stages of current LLM training pipelines. Our guiding intuition stems from the fact that the dynamics of later stages of the pipeline, e.g. post-training, can be used to inform earlier stages such as pre-training. To this end, we propose Introspective Training (or IXT), inspired by offline reward-conditioned reinforcement learning and applicable to any stage of training. IXT uses a thinking reward model to annotate data with natural language critique based feedback, enabling quality aware training from the earliest stages of the pipeline. Models are then trained by prefix-conditioning the data with the generated feedback -- ensuring that not all tokens are treated equally starting much earlier in training than usual. Comprehensive experiments on 7.5-12B transformer-based dense LLMs trained from scratch all the way up to 18 Trillion tokens seen show that our method: bends scaling curves resulting in up to 2.8x more compute efficiency generally; and reaches performance levels unachievable for models trained otherwise in domains such as math and code. |
| title | Introspective X Training: Feedback Conditioning Improves Scaling Across all LLM Training Stages |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2605.20285 |