Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.25969 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917531809218560 |
|---|---|
| author | Lin, Ke Luo, Yiyang Su, Zhaolong Song, Yunya Rao, Anyi |
| author_facet | Lin, Ke Luo, Yiyang Su, Zhaolong Song, Yunya Rao, Anyi |
| contents | Causal Transformer language models suffer from strictly sequential decoding and a quadratic per-step attention cost. While linear-time causal models and discrete diffusion models each address these weaknesses, their integration remains inherently inconsistent: diffusion requires bidirectional attention, while causal models are unidirectional. To unify these architectures, we propose $B^3D-RWKV$, a diffusion RWKV variant that integrates the model's $O(L)$ inference efficiency with parallel, bidirectional discrete-diffusion through a \emph{triplet-block layout} method. $B^3D-RWKV-7.2B$ reaches comparable accuracy on an 8-task suite versus existing models while significantly outperforming baselines in decoding throughput with an average of $\mathbf{1.6\times}$ speedup. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_25969 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Triplet-Block Diffusion RWKV Lin, Ke Luo, Yiyang Su, Zhaolong Song, Yunya Rao, Anyi Computation and Language Causal Transformer language models suffer from strictly sequential decoding and a quadratic per-step attention cost. While linear-time causal models and discrete diffusion models each address these weaknesses, their integration remains inherently inconsistent: diffusion requires bidirectional attention, while causal models are unidirectional. To unify these architectures, we propose $B^3D-RWKV$, a diffusion RWKV variant that integrates the model's $O(L)$ inference efficiency with parallel, bidirectional discrete-diffusion through a \emph{triplet-block layout} method. $B^3D-RWKV-7.2B$ reaches comparable accuracy on an 8-task suite versus existing models while significantly outperforming baselines in decoding throughput with an average of $\mathbf{1.6\times}$ speedup. |
| title | Triplet-Block Diffusion RWKV |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.25969 |