Staircase Streaming for Low-Latency Multi-Agent Inference
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908578275655680 |
|---|---|
| author | Wang, Junlin Wang, Jue Zhen Xu Athiwaratkun, Ben Dhingra, Bhuwan Zhang, Ce Zou, James |
| author_facet | Wang, Junlin Wang, Jue Zhen Xu Athiwaratkun, Ben Dhingra, Bhuwan Zhang, Ce Zou, James |
| contents | Recent advances in large language models (LLMs) opened up new directions for leveraging the collective expertise of multiple LLMs. These methods, such as Mixture-of-Agents, typically employ additional inference steps to generate intermediate outputs, which are then used to produce the final response. While multi-agent inference can enhance response quality, it can significantly increase the time to first token (TTFT), posing a challenge for latency-sensitive applications and hurting user experience. To address this issue, we propose staircase streaming for low-latency multi-agent inference. Instead of waiting for the complete intermediate outputs from previous steps, we begin generating the final response as soon as we receive partial outputs from these steps. Experimental results demonstrate that staircase streaming reduces TTFT by up to 93% while maintaining response quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_05059 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Staircase Streaming for Low-Latency Multi-Agent Inference Wang, Junlin Wang, Jue Zhen Xu Athiwaratkun, Ben Dhingra, Bhuwan Zhang, Ce Zou, James Artificial Intelligence Recent advances in large language models (LLMs) opened up new directions for leveraging the collective expertise of multiple LLMs. These methods, such as Mixture-of-Agents, typically employ additional inference steps to generate intermediate outputs, which are then used to produce the final response. While multi-agent inference can enhance response quality, it can significantly increase the time to first token (TTFT), posing a challenge for latency-sensitive applications and hurting user experience. To address this issue, we propose staircase streaming for low-latency multi-agent inference. Instead of waiting for the complete intermediate outputs from previous steps, we begin generating the final response as soon as we receive partial outputs from these steps. Experimental results demonstrate that staircase streaming reduces TTFT by up to 93% while maintaining response quality. |
| title | Staircase Streaming for Low-Latency Multi-Agent Inference |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2510.05059 |