C2T: A Classifier-Based Tree Construction Method in Speculative Decoding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929720141021184 |
|---|---|
| author | Huo, Feiye Tan, Jianchao Zhang, Kefeng Cai, Xunliang Sun, Shengli |
| author_facet | Huo, Feiye Tan, Jianchao Zhang, Kefeng Cai, Xunliang Sun, Shengli |
| contents | The growing scale of Large Language Models (LLMs) has exacerbated inference latency and computational costs. Speculative decoding methods, which aim to mitigate these issues, often face inefficiencies in the construction of token trees and the verification of candidate tokens. Existing strategies, including chain mode, static tree, and dynamic tree approaches, have limitations in accurately preparing candidate token trees for verification. We propose a novel method named C2T that adopts a lightweight classifier to generate and prune token trees dynamically. Our classifier considers additional feature variables beyond the commonly used joint probability to predict the confidence score for each draft token to determine whether it is the candidate token for verification. This method outperforms state-of-the-art (SOTA) methods such as EAGLE-2 on multiple benchmarks, by reducing the total number of candidate tokens by 25% while maintaining or even improving the acceptance length. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_13652 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | C2T: A Classifier-Based Tree Construction Method in Speculative Decoding Huo, Feiye Tan, Jianchao Zhang, Kefeng Cai, Xunliang Sun, Shengli Computation and Language Artificial Intelligence The growing scale of Large Language Models (LLMs) has exacerbated inference latency and computational costs. Speculative decoding methods, which aim to mitigate these issues, often face inefficiencies in the construction of token trees and the verification of candidate tokens. Existing strategies, including chain mode, static tree, and dynamic tree approaches, have limitations in accurately preparing candidate token trees for verification. We propose a novel method named C2T that adopts a lightweight classifier to generate and prune token trees dynamically. Our classifier considers additional feature variables beyond the commonly used joint probability to predict the confidence score for each draft token to determine whether it is the candidate token for verification. This method outperforms state-of-the-art (SOTA) methods such as EAGLE-2 on multiple benchmarks, by reducing the total number of candidate tokens by 25% while maintaining or even improving the acceptance length. |
| title | C2T: A Classifier-Based Tree Construction Method in Speculative Decoding |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.13652 |