DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912930683944960 |
|---|---|
| author | Singh, Vaibhav Ostapenko, Oleksiy Noël, Pierre-André Belilovsky, Eugene Scholak, Torsten |
| author_facet | Singh, Vaibhav Ostapenko, Oleksiy Noël, Pierre-André Belilovsky, Eugene Scholak, Torsten |
| contents | Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) generation, yet their reliance on Transformer backbones limits inference efficiency due to quadratic attention or KV-cache overhead. We introduce DiffuMamba, a masked diffusion language model built on a bidirectional Mamba backbone that combines the diffusion objective with linear-time sequence modeling, and DiffuMamba-H, a hybrid variant with interleaved attention. Across scales up to 1.3B parameters, our models match Transformer-based diffusion in downstream performance while achieving up to 8.2x and 4.3x higher inference throughput, respectively, on long sequences. We further present a systematic analysis of inference efficiency across modern DLM variants combining asymptotic complexity with empirical measurements. Notably, cache-efficient block diffusion with Mamba mixers emerges as the only strategy that scales linearly with sequence length and achieves the strongest performance across all baselines, suggesting a promising direction for future diffusion-based generation systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_15927 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone Singh, Vaibhav Ostapenko, Oleksiy Noël, Pierre-André Belilovsky, Eugene Scholak, Torsten Machine Learning Artificial Intelligence Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) generation, yet their reliance on Transformer backbones limits inference efficiency due to quadratic attention or KV-cache overhead. We introduce DiffuMamba, a masked diffusion language model built on a bidirectional Mamba backbone that combines the diffusion objective with linear-time sequence modeling, and DiffuMamba-H, a hybrid variant with interleaved attention. Across scales up to 1.3B parameters, our models match Transformer-based diffusion in downstream performance while achieving up to 8.2x and 4.3x higher inference throughput, respectively, on long sequences. We further present a systematic analysis of inference efficiency across modern DLM variants combining asymptotic complexity with empirical measurements. Notably, cache-efficient block diffusion with Mamba mixers emerges as the only strategy that scales linearly with sequence length and achieves the strongest performance across all baselines, suggesting a promising direction for future diffusion-based generation systems. |
| title | DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2511.15927 |