Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.20048 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908674014838784 |
|---|---|
| author | Huang, Zixiao Zeng, Wen Fu, Tianyu Liu, Tengxuan Sun, Yizhou Hong, Ke Yang, Xinhao Liu, Chengchun Li, Yan Zhang, Quanlu Dai, Guohao Zhu, Zhenhua Wang, Yu |
| author_facet | Huang, Zixiao Zeng, Wen Fu, Tianyu Liu, Tengxuan Sun, Yizhou Hong, Ke Yang, Xinhao Liu, Chengchun Li, Yan Zhang, Quanlu Dai, Guohao Zhu, Zhenhua Wang, Yu |
| contents | LLM-based search agents achieve strong performance but suffer from severe latency, as each step requires serialized LLM reasoning followed by action of tool execution. We revisit this bottleneck through the lens of speculation. While traditional predict-verify speculation paradigm can break serial execution, its benefit remains limited, as it retains the full original workload and adds extra inference overhead. We observe that early agent steps often involve simple evidence-gathering, where correct actions can often be predicted without full reasoning. Building on these observations, we present SPAgent, an algorithm-system co-design framework that expands the role of speculation in search agents to reduce latency. Algorithmically, SPAgent introduces a two-phase adaptive speculation mechanism that selectively omits verification when safe. System-wise, a two-level scheduler regulates speculative requests based on engine load to ensure speculation remains beneficial. We implement SPAgent in real-world systems. Across extensive experimental settings, SPAgent achieves up to $1.65\times$ end-to-end speedup while maintaining same or even achieving higher accuracy, enabling practical deployment of multi-step search agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_20048 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Reducing Latency of LLM Search Agent via Speculation-based Algorithm-System Co-Design Huang, Zixiao Zeng, Wen Fu, Tianyu Liu, Tengxuan Sun, Yizhou Hong, Ke Yang, Xinhao Liu, Chengchun Li, Yan Zhang, Quanlu Dai, Guohao Zhu, Zhenhua Wang, Yu Artificial Intelligence Machine Learning Performance LLM-based search agents achieve strong performance but suffer from severe latency, as each step requires serialized LLM reasoning followed by action of tool execution. We revisit this bottleneck through the lens of speculation. While traditional predict-verify speculation paradigm can break serial execution, its benefit remains limited, as it retains the full original workload and adds extra inference overhead. We observe that early agent steps often involve simple evidence-gathering, where correct actions can often be predicted without full reasoning. Building on these observations, we present SPAgent, an algorithm-system co-design framework that expands the role of speculation in search agents to reduce latency. Algorithmically, SPAgent introduces a two-phase adaptive speculation mechanism that selectively omits verification when safe. System-wise, a two-level scheduler regulates speculative requests based on engine load to ensure speculation remains beneficial. We implement SPAgent in real-world systems. Across extensive experimental settings, SPAgent achieves up to $1.65\times$ end-to-end speedup while maintaining same or even achieving higher accuracy, enabling practical deployment of multi-step search agents. |
| title | Reducing Latency of LLM Search Agent via Speculation-based Algorithm-System Co-Design |
| topic | Artificial Intelligence Machine Learning Performance |
| url | https://arxiv.org/abs/2511.20048 |