BitLM: Unlocking Multi-Token Language Generation with Bitwise Continuous Diffusion
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916003919691776 |
|---|---|
| author | Zhuang, Shaobin Ai, Yuang Han, Jiaming Li, Xiaohui Huang, Huaibo Yue, Xiangyu Hu, Xuefeng Xu, Kun Wang, Yali Chen, Hao |
| author_facet | Zhuang, Shaobin Ai, Yuang Han, Jiaming Li, Xiaohui Huang, Huaibo Yue, Xiangyu Hu, Xuefeng Xu, Kun Wang, Yali Chen, Hao |
| contents | Autoregressive language models generate text one token at a time, yet natural language is inherently structured in multi-token units, including phrases, n-grams, and collocations that carry meaning jointly. This one-token bottleneck limits both the expressiveness of the model during pre-training and its throughput at inference time. Existing remedies such as speculative decoding or diffusion-based language models either leave the underlying bottleneck intact or sacrifice the causal structure essential to language modeling. We propose BitLM, a language model that represents each token as a fixed-length binary code and employs a lightweight diffusion head to denoise multiple tokens in parallel within each block. Crucially, BitLM preserves left-to-right causal attention across blocks while making joint lexical decisions within each block, combining the reliability of autoregressive modeling with the parallelism of iterative refinement. By replacing the large-vocabulary softmax with bitwise denoising, BitLM reframes token generation as iterative commitment in a compact binary space, enabling more efficient pre-training and substantially faster inference without altering the causal foundation that makes language models effective. Our results demonstrate that the one-token-at-a-time paradigm is not a fundamental requirement but an interface choice, and that changing it can yield a stronger and faster language model. We hope BitLM points toward a promising direction for next-generation language model architectures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_11577 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | BitLM: Unlocking Multi-Token Language Generation with Bitwise Continuous Diffusion Zhuang, Shaobin Ai, Yuang Han, Jiaming Li, Xiaohui Huang, Huaibo Yue, Xiangyu Hu, Xuefeng Xu, Kun Wang, Yali Chen, Hao Computation and Language Autoregressive language models generate text one token at a time, yet natural language is inherently structured in multi-token units, including phrases, n-grams, and collocations that carry meaning jointly. This one-token bottleneck limits both the expressiveness of the model during pre-training and its throughput at inference time. Existing remedies such as speculative decoding or diffusion-based language models either leave the underlying bottleneck intact or sacrifice the causal structure essential to language modeling. We propose BitLM, a language model that represents each token as a fixed-length binary code and employs a lightweight diffusion head to denoise multiple tokens in parallel within each block. Crucially, BitLM preserves left-to-right causal attention across blocks while making joint lexical decisions within each block, combining the reliability of autoregressive modeling with the parallelism of iterative refinement. By replacing the large-vocabulary softmax with bitwise denoising, BitLM reframes token generation as iterative commitment in a compact binary space, enabling more efficient pre-training and substantially faster inference without altering the causal foundation that makes language models effective. Our results demonstrate that the one-token-at-a-time paradigm is not a fundamental requirement but an interface choice, and that changing it can yield a stronger and faster language model. We hope BitLM points toward a promising direction for next-generation language model architectures. |
| title | BitLM: Unlocking Multi-Token Language Generation with Bitwise Continuous Diffusion |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.11577 |