Consultant Decoding: Yet Another Synergistic Mechanism
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912410949910528 |
|---|---|
| author | Ding, Chuanghao Wang, Jiaping Yang, Ziqing Wang, Xiaoliang Lin, Dahua Nguyen, Cam-Tu Tan, Fei |
| author_facet | Ding, Chuanghao Wang, Jiaping Yang, Ziqing Wang, Xiaoliang Lin, Dahua Nguyen, Cam-Tu Tan, Fei |
| contents | The synergistic mechanism based on Speculative Decoding (SD) has garnered considerable attention as a simple yet effective approach for accelerating the inference of large language models (LLMs). Nonetheless, the high rejection rates require repeated LLMs calls to validate draft tokens, undermining the overall efficiency gain of SD. In this work, we revisit existing verification mechanisms and propose a novel synergetic mechanism Consultant Decoding (CD). Unlike SD, which relies on a metric derived from importance sampling for verification, CD verifies candidate drafts using token-level likelihoods computed solely by the LLM. CD achieves up to a 2.5-fold increase in inference speed compared to the target model, while maintaining comparable generation quality (around 100% of the target model's performance). Interestingly, this is achieved by combining models whose parameter sizes differ by two orders of magnitude. In addition, CD reduces the call frequency of the large target model to below 10%, particularly in more demanding tasks. CD's performance was even found to surpass that of the large target model, which theoretically represents the upper bound for speculative decoding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_02391 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Consultant Decoding: Yet Another Synergistic Mechanism Ding, Chuanghao Wang, Jiaping Yang, Ziqing Wang, Xiaoliang Lin, Dahua Nguyen, Cam-Tu Tan, Fei Computation and Language Artificial Intelligence The synergistic mechanism based on Speculative Decoding (SD) has garnered considerable attention as a simple yet effective approach for accelerating the inference of large language models (LLMs). Nonetheless, the high rejection rates require repeated LLMs calls to validate draft tokens, undermining the overall efficiency gain of SD. In this work, we revisit existing verification mechanisms and propose a novel synergetic mechanism Consultant Decoding (CD). Unlike SD, which relies on a metric derived from importance sampling for verification, CD verifies candidate drafts using token-level likelihoods computed solely by the LLM. CD achieves up to a 2.5-fold increase in inference speed compared to the target model, while maintaining comparable generation quality (around 100% of the target model's performance). Interestingly, this is achieved by combining models whose parameter sizes differ by two orders of magnitude. In addition, CD reduces the call frequency of the large target model to below 10%, particularly in more demanding tasks. CD's performance was even found to surpass that of the large target model, which theoretically represents the upper bound for speculative decoding. |
| title | Consultant Decoding: Yet Another Synergistic Mechanism |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2506.02391 |