Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.04344 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910194138611712 |
|---|---|
| author | Cen, Zetai Zhu, Jin Shen, Xinwei Shi, Chengchun |
| author_facet | Cen, Zetai Zhu, Jin Shen, Xinwei Shi, Chengchun |
| contents | We introduce a simple yet powerful framework for training large language models. In contrast to the standard autoregressive next-token prediction based on an exact prefix, we propose a perturbation-based procedure that first transforms the prefix into a semantic neighbor and then conditions on this perturbed variant for next-token prediction. This yields a hierarchical model with a pre-post-additive noise structure. Within this framework, we develop a rigorous theory of extrapolability, namely, the capacity of a model class to make reliable predictions for token sequences that lie outside the empirical support of the training corpus. We evaluate the finite-sample performance of the proposed procedure using both synthetic and real-world language data. Results show that the proposed method consistently improves out-of-support prediction while maintaining competitive in-support performance, demonstrating that perturbation offers a practical route to language modeling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_04344 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Perturbation is All You Need for Extrapolating Language Models Cen, Zetai Zhu, Jin Shen, Xinwei Shi, Chengchun Machine Learning Statistics Theory We introduce a simple yet powerful framework for training large language models. In contrast to the standard autoregressive next-token prediction based on an exact prefix, we propose a perturbation-based procedure that first transforms the prefix into a semantic neighbor and then conditions on this perturbed variant for next-token prediction. This yields a hierarchical model with a pre-post-additive noise structure. Within this framework, we develop a rigorous theory of extrapolability, namely, the capacity of a model class to make reliable predictions for token sequences that lie outside the empirical support of the training corpus. We evaluate the finite-sample performance of the proposed procedure using both synthetic and real-world language data. Results show that the proposed method consistently improves out-of-support prediction while maintaining competitive in-support performance, demonstrating that perturbation offers a practical route to language modeling. |
| title | Perturbation is All You Need for Extrapolating Language Models |
| topic | Machine Learning Statistics Theory |
| url | https://arxiv.org/abs/2605.04344 |