LEDOM: Reverse Language Model
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918366952816640 |
|---|---|
| author | Yin, Xunjian Cheng, Sitao Xie, Yuxi Hu, Xinyu Lin, Li Wang, Xinyi Pan, Liangming Wang, William Yang Wan, Xiaojun |
| author_facet | Yin, Xunjian Cheng, Sitao Xie, Yuxi Hu, Xinyu Lin, Li Wang, Xinyi Pan, Liangming Wang, William Yang Wan, Xiaojun |
| contents | Autoregressive language models are trained exclusively left-to-right. We explore the complementary factorization, training right-to-left at scale, and ask what reasoning patterns emerge when a model conditions on future context to predict the past. We train LEDOM, an open-source purely reverse autoregressive language model (2B/7B parameters, 435B tokens), and find it develops capabilities distinct from forward models, including abductive inference, question synthesis, and natural resolution of the reversal curse. We then explore one application of the reverse model: combining forward likelihood $P(y \mid x)$ with reverse posterior $P(x \mid y)$ through noisy channel duality. We propose Reverse Reward, which reranks forward outputs using reverse posterior estimates, and prove that bidirectional scoring penalizes hallucinated reasoning chains whose backward reconstruction degrades. Reverse Reward yields gains of up to 6.6\% on AIME 2024 and 15\% on AMC 2023 across multiple strong baselines. We release all models, code, and data here. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_01335 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LEDOM: Reverse Language Model Yin, Xunjian Cheng, Sitao Xie, Yuxi Hu, Xinyu Lin, Li Wang, Xinyi Pan, Liangming Wang, William Yang Wan, Xiaojun Computation and Language Artificial Intelligence Autoregressive language models are trained exclusively left-to-right. We explore the complementary factorization, training right-to-left at scale, and ask what reasoning patterns emerge when a model conditions on future context to predict the past. We train LEDOM, an open-source purely reverse autoregressive language model (2B/7B parameters, 435B tokens), and find it develops capabilities distinct from forward models, including abductive inference, question synthesis, and natural resolution of the reversal curse. We then explore one application of the reverse model: combining forward likelihood $P(y \mid x)$ with reverse posterior $P(x \mid y)$ through noisy channel duality. We propose Reverse Reward, which reranks forward outputs using reverse posterior estimates, and prove that bidirectional scoring penalizes hallucinated reasoning chains whose backward reconstruction degrades. Reverse Reward yields gains of up to 6.6\% on AIME 2024 and 15\% on AMC 2023 across multiple strong baselines. We release all models, code, and data here. |
| title | LEDOM: Reverse Language Model |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2507.01335 |