Dynamic Depth Decoding: Faster Speculative Decoding for LLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910585937985536 |
|---|---|
| author | Brown, Oscar Wang, Zhengjie Do, Andrea Mathew, Nikhil Yu, Cheng |
| author_facet | Brown, Oscar Wang, Zhengjie Do, Andrea Mathew, Nikhil Yu, Cheng |
| contents | The acceleration of Large Language Models (LLMs) with speculative decoding provides a significant runtime improvement without any loss of accuracy. Currently, EAGLE-2 is the state-of-the-art speculative decoding method, improving on EAGLE with a dynamic draft tree. We introduce Dynamic Depth Decoding (DDD), which optimises EAGLE-2's tree drafting method using a dynamic depth. This extends the average speedup that EAGLE-2 achieves over EAGLE by $44\%$, giving DDD an average speedup of $3.16$x. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_00142 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Dynamic Depth Decoding: Faster Speculative Decoding for LLMs Brown, Oscar Wang, Zhengjie Do, Andrea Mathew, Nikhil Yu, Cheng Computation and Language Artificial Intelligence The acceleration of Large Language Models (LLMs) with speculative decoding provides a significant runtime improvement without any loss of accuracy. Currently, EAGLE-2 is the state-of-the-art speculative decoding method, improving on EAGLE with a dynamic draft tree. We introduce Dynamic Depth Decoding (DDD), which optimises EAGLE-2's tree drafting method using a dynamic depth. This extends the average speedup that EAGLE-2 achieves over EAGLE by $44\%$, giving DDD an average speedup of $3.16$x. |
| title | Dynamic Depth Decoding: Faster Speculative Decoding for LLMs |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2409.00142 |