Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.15148 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918352229761024 |
|---|---|
| author | Xiong, Jing Chen, Qiujiang Ye, Fanghua Wan, Zhongwei Zheng, Chuanyang Zhao, Chenyang Shen, Hui Li, Hanbo Tao, Chaofan Tan, Haochen Bai, Haoli Shang, Lifeng Kong, Lingpeng Wong, Ngai |
| author_facet | Xiong, Jing Chen, Qiujiang Ye, Fanghua Wan, Zhongwei Zheng, Chuanyang Zhao, Chenyang Shen, Hui Li, Hanbo Tao, Chaofan Tan, Haochen Bai, Haoli Shang, Lifeng Kong, Lingpeng Wong, Ngai |
| contents | Large language models (LLMs) benefit from test-time scaling but are often hampered by high inference latency. Speculative decoding is a natural way to accelerate the scaling process; however, scaling along both the parallel and sequential dimensions poses significant challenges, including substantial memory-bound execution and synchronization overhead. We introduce ATTS (Asynchronous Test-Time Scaling), a statistically guaranteed adaptive scaling framework that follows the hypothesis testing process to address these challenges. By revisiting arithmetic intensity, ATTS identifies synchronization as the primary bottleneck. It enables asynchronous inference through online calibration and proposes an ordinal classification algorithm that supports a three-stage rejection sampling pipeline, scaling along both the sequential and parallel axes. Across experiments on the MATH, AMC23, AIME24, and AIME25 datasets and across multiple draft-target model families, we show that ATTS delivers up to 56.7x speedup in test-time scaling and a 4.14x throughput improvement, while maintaining accurate control of the rejection rate, reducing latency and memory overhead, and incurring no accuracy loss. By scaling both in parallel and sequential dimensions, we enable the 1.5B/70B draft/target model combination to achieve the performance of the state-of-the-art reasoning model o3-mini (high) on the AIME dataset. We have released the code at https://github.com/menik1126/asynchronous-test-time-scaling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_15148 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ATTS: Asynchronous Test-Time Scaling via Conformal Prediction Xiong, Jing Chen, Qiujiang Ye, Fanghua Wan, Zhongwei Zheng, Chuanyang Zhao, Chenyang Shen, Hui Li, Hanbo Tao, Chaofan Tan, Haochen Bai, Haoli Shang, Lifeng Kong, Lingpeng Wong, Ngai Computation and Language Large language models (LLMs) benefit from test-time scaling but are often hampered by high inference latency. Speculative decoding is a natural way to accelerate the scaling process; however, scaling along both the parallel and sequential dimensions poses significant challenges, including substantial memory-bound execution and synchronization overhead. We introduce ATTS (Asynchronous Test-Time Scaling), a statistically guaranteed adaptive scaling framework that follows the hypothesis testing process to address these challenges. By revisiting arithmetic intensity, ATTS identifies synchronization as the primary bottleneck. It enables asynchronous inference through online calibration and proposes an ordinal classification algorithm that supports a three-stage rejection sampling pipeline, scaling along both the sequential and parallel axes. Across experiments on the MATH, AMC23, AIME24, and AIME25 datasets and across multiple draft-target model families, we show that ATTS delivers up to 56.7x speedup in test-time scaling and a 4.14x throughput improvement, while maintaining accurate control of the rejection rate, reducing latency and memory overhead, and incurring no accuracy loss. By scaling both in parallel and sequential dimensions, we enable the 1.5B/70B draft/target model combination to achieve the performance of the state-of-the-art reasoning model o3-mini (high) on the AIME dataset. We have released the code at https://github.com/menik1126/asynchronous-test-time-scaling. |
| title | ATTS: Asynchronous Test-Time Scaling via Conformal Prediction |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2509.15148 |