OWL: Overcoming Window Length-Dependence in Speculative Decoding for Long-Context Inputs
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916997881659392 |
|---|---|
| author | Lee, Jaeseong hwang, seung-won Qiao, Aurick Oliaro, Gabriele Wang, Ye Rajbhandari, Samyam |
| author_facet | Lee, Jaeseong hwang, seung-won Qiao, Aurick Oliaro, Gabriele Wang, Ye Rajbhandari, Samyam |
| contents | Speculative decoding promises faster inference for large language models (LLMs), yet existing methods fail to generalize to real-world settings. Benchmarks typically assume short contexts (e.g., 2K tokens), whereas practical workloads involve long contexts. We find current approaches degrade severely with long contexts; for instance, EAGLE3 even slows down the generation speed by 0.81x. We address these limitations by releasing a new long-context benchmark (LongSpecBench) and introducing a novel model (OWL). OWL achieves about 5x higher acceptance length than EAGLE3 on long-context inputs through three innovations: (1) an LSTM-based drafter conditioned only on the last-token state, making it generalize to various lengths, (2) a special token [SPEC] in the verifier that produces richer representation for drafter, and (3) a hybrid algorithm combining both tree and non-tree decoding methods. We release all code and datasets to advance future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_07535 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | OWL: Overcoming Window Length-Dependence in Speculative Decoding for Long-Context Inputs Lee, Jaeseong hwang, seung-won Qiao, Aurick Oliaro, Gabriele Wang, Ye Rajbhandari, Samyam Computation and Language Artificial Intelligence Speculative decoding promises faster inference for large language models (LLMs), yet existing methods fail to generalize to real-world settings. Benchmarks typically assume short contexts (e.g., 2K tokens), whereas practical workloads involve long contexts. We find current approaches degrade severely with long contexts; for instance, EAGLE3 even slows down the generation speed by 0.81x. We address these limitations by releasing a new long-context benchmark (LongSpecBench) and introducing a novel model (OWL). OWL achieves about 5x higher acceptance length than EAGLE3 on long-context inputs through three innovations: (1) an LSTM-based drafter conditioned only on the last-token state, making it generalize to various lengths, (2) a special token [SPEC] in the verifier that produces richer representation for drafter, and (3) a hybrid algorithm combining both tree and non-tree decoding methods. We release all code and datasets to advance future research. |
| title | OWL: Overcoming Window Length-Dependence in Speculative Decoding for Long-Context Inputs |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.07535 |