SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Serving
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914364908371968 |
|---|---|
| author | Woo, Sunghyeon Seo, Ahreum Lee, Jaegwang Kil, Jaeeun Seo, Hanbae Kim, Joonghoon Park, Baeseong Kwon, Se Jung Lee, Dongsoo |
| author_facet | Woo, Sunghyeon Seo, Ahreum Lee, Jaegwang Kil, Jaeeun Seo, Hanbae Kim, Joonghoon Park, Baeseong Kwon, Se Jung Lee, Dongsoo |
| contents | In multi-model LLM serving, decode execution remains inefficient due to model-specific resource partitioning: since cross-model batching is not possible, memory-bound decoding often suffers from severe GPU underutilization, especially under skewed workloads. We propose Shared Use of Next-token Prediction (SUN), the first approach that enables cross-model sharing of decode execution in disaggregated multi-LLM serving. SUN decomposes a decoder-only Transformer into a prefill module and a decode module, and fine-tunes only the task-specific prefill module, enabling a frozen decode module to be shared across models. This design enables a model-agnostic decode routing policy that balances decode requests across shared workers to maximize utilization. Across diverse tasks and model families, SUN achieves accuracy comparable to full fine-tuning while maintaining system throughput with fewer decode workers. In particular, SUN improves throughput per GPU by up to 2.0x over conventional disaggregation while keeping time-per-output-token (TPOT) within 5%. SUN inherently enables and facilitates low-bit decoding; with Quantized SUN (QSUN), it achieves a 45% speedup with comparable accuracy to SUN while preserving the benefits of shared decoding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_02599 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Serving Woo, Sunghyeon Seo, Ahreum Lee, Jaegwang Kil, Jaeeun Seo, Hanbae Kim, Joonghoon Park, Baeseong Kwon, Se Jung Lee, Dongsoo Artificial Intelligence Machine Learning In multi-model LLM serving, decode execution remains inefficient due to model-specific resource partitioning: since cross-model batching is not possible, memory-bound decoding often suffers from severe GPU underutilization, especially under skewed workloads. We propose Shared Use of Next-token Prediction (SUN), the first approach that enables cross-model sharing of decode execution in disaggregated multi-LLM serving. SUN decomposes a decoder-only Transformer into a prefill module and a decode module, and fine-tunes only the task-specific prefill module, enabling a frozen decode module to be shared across models. This design enables a model-agnostic decode routing policy that balances decode requests across shared workers to maximize utilization. Across diverse tasks and model families, SUN achieves accuracy comparable to full fine-tuning while maintaining system throughput with fewer decode workers. In particular, SUN improves throughput per GPU by up to 2.0x over conventional disaggregation while keeping time-per-output-token (TPOT) within 5%. SUN inherently enables and facilitates low-bit decoding; with Quantized SUN (QSUN), it achieves a 45% speedup with comparable accuracy to SUN while preserving the benefits of shared decoding. |
| title | SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Serving |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2603.02599 |