Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.13284 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916009025208320 |
|---|---|
| author | Cen, Zetai Gu, Chenfei Zhu, Jin Li, Ting Chen, Yunxiao Shi, Chengchun |
| author_facet | Cen, Zetai Gu, Chenfei Zhu, Jin Li, Ting Chen, Yunxiao Shi, Chengchun |
| contents | Recent advancements in large language models demonstrate that injecting perturbations can substantially enhance extrapolation performance. However, current approaches often rely on discrete perturbations with fixed designs, which limits their flexibility. In this work, we propose a framework where token prefixes are perturbed by a learnable transformation of a continuous latent vector within an embedding space. To overcome the challenge of an intractable marginal likelihood, we derive unbiased estimating equations for model parameters and optimize them via stochastic gradient descent. We establish the statistical properties of the resulting estimator in over-parameterized regimes. Empirical evaluations on both synthetic and real-world datasets demonstrate that our proposal yields significant gains in out-of-domain settings over a range of state-of-the-art baseline methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_13284 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Learning Perturbations to Extrapolate Your LLM Cen, Zetai Gu, Chenfei Zhu, Jin Li, Ting Chen, Yunxiao Shi, Chengchun Machine Learning Statistics Theory Recent advancements in large language models demonstrate that injecting perturbations can substantially enhance extrapolation performance. However, current approaches often rely on discrete perturbations with fixed designs, which limits their flexibility. In this work, we propose a framework where token prefixes are perturbed by a learnable transformation of a continuous latent vector within an embedding space. To overcome the challenge of an intractable marginal likelihood, we derive unbiased estimating equations for model parameters and optimize them via stochastic gradient descent. We establish the statistical properties of the resulting estimator in over-parameterized regimes. Empirical evaluations on both synthetic and real-world datasets demonstrate that our proposal yields significant gains in out-of-domain settings over a range of state-of-the-art baseline methods. |
| title | Learning Perturbations to Extrapolate Your LLM |
| topic | Machine Learning Statistics Theory |
| url | https://arxiv.org/abs/2605.13284 |