Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.07347 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908759545085952 |
|---|---|
| author | He, Shaokai Wei, Kaiwen Zeng, Xinyi Chen, Xiang Yang, Xue Li, Zhenyang Zhong, Jiang Tian, Yu |
| author_facet | He, Shaokai Wei, Kaiwen Zeng, Xinyi Chen, Xiang Yang, Xue Li, Zhenyang Zhong, Jiang Tian, Yu |
| contents | The "reversal curse" refers to the phenomenon where large language models (LLMs) exhibit predominantly unidirectional behavior when processing logically bidirectional relationships. Prior work attributed this to autoregressive training -- predicting the next token inherently favors left-to-right information flow over genuine bidirectional knowledge associations. However, we observe that Diffusion LLMs (DLLMs), despite being trained bidirectionally, also suffer from the reversal curse. To investigate the root causes, we conduct systematic experiments on DLLMs and identify three key reasons: 1) entity fragmentation during training, 2) data asymmetry, and 3) missing entity relations. Motivated by the analysis of these reasons, we propose Diffusion Entity-Relation Modeling (DiffER), which addresses the reversal curse through entity-aware training and balanced data construction. Specifically, DiffER introduces whole-entity masking, which mitigates entity fragmentation by predicting complete entities in a single step. DiffER further employs distribution-symmetric and relation-enhanced data construction strategies to alleviate data asymmetry and missing relations. Extensive experiments demonstrate that DiffER effectively alleviates the reversal curse in Diffusion LLMs, offering new perspectives for future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_07347 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | DiffER: Diffusion Entity-Relation Modeling for Reversal Curse in Diffusion Large Language Models He, Shaokai Wei, Kaiwen Zeng, Xinyi Chen, Xiang Yang, Xue Li, Zhenyang Zhong, Jiang Tian, Yu Computation and Language The "reversal curse" refers to the phenomenon where large language models (LLMs) exhibit predominantly unidirectional behavior when processing logically bidirectional relationships. Prior work attributed this to autoregressive training -- predicting the next token inherently favors left-to-right information flow over genuine bidirectional knowledge associations. However, we observe that Diffusion LLMs (DLLMs), despite being trained bidirectionally, also suffer from the reversal curse. To investigate the root causes, we conduct systematic experiments on DLLMs and identify three key reasons: 1) entity fragmentation during training, 2) data asymmetry, and 3) missing entity relations. Motivated by the analysis of these reasons, we propose Diffusion Entity-Relation Modeling (DiffER), which addresses the reversal curse through entity-aware training and balanced data construction. Specifically, DiffER introduces whole-entity masking, which mitigates entity fragmentation by predicting complete entities in a single step. DiffER further employs distribution-symmetric and relation-enhanced data construction strategies to alleviate data asymmetry and missing relations. Extensive experiments demonstrate that DiffER effectively alleviates the reversal curse in Diffusion LLMs, offering new perspectives for future research. |
| title | DiffER: Diffusion Entity-Relation Modeling for Reversal Curse in Diffusion Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.07347 |