SEED: Accelerating Reasoning Tree Construction via Scheduled Speculative Decoding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915068103360512 |
|---|---|
| author | Wang, Zhenglin Wu, Jialong Lai, Yilong Zhang, Congzhi Zhou, Deyu |
| author_facet | Wang, Zhenglin Wu, Jialong Lai, Yilong Zhang, Congzhi Zhou, Deyu |
| contents | Large Language Models (LLMs) demonstrate remarkable emergent abilities across various tasks, yet fall short of complex reasoning and planning tasks. The tree-search-based reasoning methods address this by surpassing the capabilities of chain-of-thought prompting, encouraging exploration of intermediate steps. However, such methods introduce significant inference latency due to the systematic exploration and evaluation of multiple thought paths. This paper introduces SeeD, a novel and efficient inference framework to optimize runtime speed and GPU memory management concurrently. By employing a scheduled speculative execution, SeeD efficiently handles multiple iterations for the thought generation and the state evaluation, leveraging a rounds-scheduled strategy to manage draft model dispatching. Extensive experimental evaluations on three reasoning datasets demonstrate superior speedup performance of SeeD, providing a viable path for batched inference in training-free speculative decoding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_18200 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SEED: Accelerating Reasoning Tree Construction via Scheduled Speculative Decoding Wang, Zhenglin Wu, Jialong Lai, Yilong Zhang, Congzhi Zhou, Deyu Computation and Language Large Language Models (LLMs) demonstrate remarkable emergent abilities across various tasks, yet fall short of complex reasoning and planning tasks. The tree-search-based reasoning methods address this by surpassing the capabilities of chain-of-thought prompting, encouraging exploration of intermediate steps. However, such methods introduce significant inference latency due to the systematic exploration and evaluation of multiple thought paths. This paper introduces SeeD, a novel and efficient inference framework to optimize runtime speed and GPU memory management concurrently. By employing a scheduled speculative execution, SeeD efficiently handles multiple iterations for the thought generation and the state evaluation, leveraging a rounds-scheduled strategy to manage draft model dispatching. Extensive experimental evaluations on three reasoning datasets demonstrate superior speedup performance of SeeD, providing a viable path for batched inference in training-free speculative decoding. |
| title | SEED: Accelerating Reasoning Tree Construction via Scheduled Speculative Decoding |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.18200 |