Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.11342 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909992539389952 |
|---|---|
| author | Yu, Chuanyue Wang, Jiahui Li, Yuhan Chang, Heng Lan, Ge Sun, Qingyun Li, Jia Li, Jianxin Zhang, Ziwei |
| author_facet | Yu, Chuanyue Wang, Jiahui Li, Yuhan Chang, Heng Lan, Ge Sun, Qingyun Li, Jia Li, Jianxin Zhang, Ziwei |
| contents | Diffusion Language Models (DLMs) have recently demonstrated remarkable capabilities in natural language processing tasks. However, the potential of Retrieval-Augmented Generation (RAG), which shows great successes for enhancing large language models (LLMs), has not been well explored, due to the fundamental difference between LLM and DLM decoding. To fill this critical gap, we systematically test the performance of DLMs within the RAG framework. Our findings reveal that DLMs coupled with RAG show promising potentials with stronger dependency on contextual information, but suffer from limited generation precision. We identify a key underlying issue: Response Semantic Drift (RSD), where the generated answer progressively deviates from the query's original semantics, leading to low precision content. We trace this problem to the denoising strategies in DLMs, which fail to maintain semantic alignment with the query throughout the iterative denoising process. To address this, we propose Semantic-Preserving REtrieval-Augmented Diffusion (SPREAD), a novel framework that introduces a query-relevance-guided denoising strategy. By actively guiding the denoising trajectory, SPREAD ensures the generation remains anchored to the query's semantics and effectively suppresses drift. Experimental results demonstrate that SPREAD significantly enhances the precision and effectively mitigates RSD of generated answers within the RAG framework. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_11342 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Unlocking the Potentials of Retrieval-Augmented Generation for Diffusion Language Models Yu, Chuanyue Wang, Jiahui Li, Yuhan Chang, Heng Lan, Ge Sun, Qingyun Li, Jia Li, Jianxin Zhang, Ziwei Machine Learning Computation and Language Diffusion Language Models (DLMs) have recently demonstrated remarkable capabilities in natural language processing tasks. However, the potential of Retrieval-Augmented Generation (RAG), which shows great successes for enhancing large language models (LLMs), has not been well explored, due to the fundamental difference between LLM and DLM decoding. To fill this critical gap, we systematically test the performance of DLMs within the RAG framework. Our findings reveal that DLMs coupled with RAG show promising potentials with stronger dependency on contextual information, but suffer from limited generation precision. We identify a key underlying issue: Response Semantic Drift (RSD), where the generated answer progressively deviates from the query's original semantics, leading to low precision content. We trace this problem to the denoising strategies in DLMs, which fail to maintain semantic alignment with the query throughout the iterative denoising process. To address this, we propose Semantic-Preserving REtrieval-Augmented Diffusion (SPREAD), a novel framework that introduces a query-relevance-guided denoising strategy. By actively guiding the denoising trajectory, SPREAD ensures the generation remains anchored to the query's semantics and effectively suppresses drift. Experimental results demonstrate that SPREAD significantly enhances the precision and effectively mitigates RSD of generated answers within the RAG framework. |
| title | Unlocking the Potentials of Retrieval-Augmented Generation for Diffusion Language Models |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2601.11342 |