Efficiently Scaling LLM Reasoning with Certaindex
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912396839223296 |
|---|---|
| author | Fu, Yichao Chen, Junda Zhu, Siqi Fu, Zheyu Dai, Zhongdongming Zhuang, Yonghao Ma, Yian Qiao, Aurick Rosing, Tajana Stoica, Ion Zhang, Hao |
| author_facet | Fu, Yichao Chen, Junda Zhu, Siqi Fu, Zheyu Dai, Zhongdongming Zhuang, Yonghao Ma, Yian Qiao, Aurick Rosing, Tajana Stoica, Ion Zhang, Hao |
| contents | Test-time reasoning algorithms such as chain-of-thought, self-consistency, and MCTS enhance LLM problem-solving but can wastefully generate many tokens without improving accuracy. At the same time, we observe that these algorithms exhibit answer stabilization: their intermediate solutions often cease to change after a certain point, and further investment of compute does not change their final answer. To quantify this phenomenon, we introduce Certaindex, an algorithm-agnostic metric measuring this evolving stability, signaling when further computation is unlikely to alter the final result. Certaindex is lightweight, can accelerate reasoning program inference via early exit, and further enables dynamic token allocation, gang scheduling, and many opportunities when integrated with real-world LLM serving systems. To quantify real-world benefits, we built Certaindex as a scheduler into Dynasor, our reasoning-aware LLM serving system, and demonstrate up to 50% compute savings and 3.3x higher throughput in real workloads with no accuracy drop. Our code is available at https://github.com/hao-ai-lab/Dynasor.git |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_20993 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Efficiently Scaling LLM Reasoning with Certaindex Fu, Yichao Chen, Junda Zhu, Siqi Fu, Zheyu Dai, Zhongdongming Zhuang, Yonghao Ma, Yian Qiao, Aurick Rosing, Tajana Stoica, Ion Zhang, Hao Machine Learning Computation and Language Test-time reasoning algorithms such as chain-of-thought, self-consistency, and MCTS enhance LLM problem-solving but can wastefully generate many tokens without improving accuracy. At the same time, we observe that these algorithms exhibit answer stabilization: their intermediate solutions often cease to change after a certain point, and further investment of compute does not change their final answer. To quantify this phenomenon, we introduce Certaindex, an algorithm-agnostic metric measuring this evolving stability, signaling when further computation is unlikely to alter the final result. Certaindex is lightweight, can accelerate reasoning program inference via early exit, and further enables dynamic token allocation, gang scheduling, and many opportunities when integrated with real-world LLM serving systems. To quantify real-world benefits, we built Certaindex as a scheduler into Dynasor, our reasoning-aware LLM serving system, and demonstrate up to 50% compute savings and 3.3x higher throughput in real workloads with no accuracy drop. Our code is available at https://github.com/hao-ai-lab/Dynasor.git |
| title | Efficiently Scaling LLM Reasoning with Certaindex |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2412.20993 |