Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2512.07173 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917414922354688 |
|---|---|
| author | Shen, Jucheng Sarkar, Gaurav Ro, Yeonju Sridhar, Sharath Nittur Wang, Zhangyang Akella, Aditya Kundu, Souvik |
| author_facet | Shen, Jucheng Sarkar, Gaurav Ro, Yeonju Sridhar, Sharath Nittur Wang, Zhangyang Akella, Aditya Kundu, Souvik |
| contents | We present CadLLM, a training-free method to accelerate the inference throughput of diffusion-based LLMs (dLLMs). We first investigate the dynamic nature of token unmasking confidence across blocks and steps. Based on this observation, we present a lightweight adaptive approach that controls the generation block size, step size, and threshold based on the average confidence of unmasked tokens. We further reduce softmax overhead by dynamically leveraging a subset of the vocabulary to regulate sampling breadth. CadLLM is a plug-and-play, model-agnostic method compatible with KV-cache-based dLLMs. Extensive experiments on four popular tasks demonstrate that CadLLM yields up to 1.1-2.28x throughput improvement over the state-of-the-art baseline with competitive accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_07173 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Improving the Throughput of Diffusion-based Large Language Models via a Training-Free Confidence-Aware Calibration Shen, Jucheng Sarkar, Gaurav Ro, Yeonju Sridhar, Sharath Nittur Wang, Zhangyang Akella, Aditya Kundu, Souvik Machine Learning We present CadLLM, a training-free method to accelerate the inference throughput of diffusion-based LLMs (dLLMs). We first investigate the dynamic nature of token unmasking confidence across blocks and steps. Based on this observation, we present a lightweight adaptive approach that controls the generation block size, step size, and threshold based on the average confidence of unmasked tokens. We further reduce softmax overhead by dynamically leveraging a subset of the vocabulary to regulate sampling breadth. CadLLM is a plug-and-play, model-agnostic method compatible with KV-cache-based dLLMs. Extensive experiments on four popular tasks demonstrate that CadLLM yields up to 1.1-2.28x throughput improvement over the state-of-the-art baseline with competitive accuracy. |
| title | Improving the Throughput of Diffusion-based Large Language Models via a Training-Free Confidence-Aware Calibration |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2512.07173 |