Efficiently Serving Large Multimodal Models Using EPD Disaggregation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915363269115904 |
|---|---|
| author | Singh, Gursimran Wang, Xinglu Hu, Yifan Yu, Timothy Xing, Linzi Jiang, Wei Wang, Zhefeng Bai, Xiaolong Li, Yi Xiong, Ying Zhang, Yong Fan, Zhenan |
| author_facet | Singh, Gursimran Wang, Xinglu Hu, Yifan Yu, Timothy Xing, Linzi Jiang, Wei Wang, Zhefeng Bai, Xiaolong Li, Yi Xiong, Ying Zhang, Yong Fan, Zhenan |
| contents | Large Multimodal Models (LMMs) extend Large Language Models (LLMs) by handling diverse inputs such as images, audio, and video, but at the cost of adding a multimodal encoding stage that increases both computational and memory overhead. This step negatively affects key Service Level Objectives (SLOs), such as time to first token (TTFT) and time per output token (TPOT). We introduce Encode-Prefill-Decode (EPD) Disaggregation, a novel framework that separates the encoding, prefill, and decode stages onto dedicated resources. Unlike current systems, which bundle encoding and prefill together, our approach decouples these steps, unlocking new opportunities and optimizations. These include a mechanism to cache multimedia tokens for efficient transfer, a novel way to parallelize the encoding load within a request, a module for optimal resource allocation for disaggregated serving, and a novel role-switching method to handle changing workload characteristics. Experimental evaluations with popular LMMs show substantial gains in memory efficiency (up to 15x lower peak memory utilization), batch sizes (up to 22x larger), 10x more images per request, and 2.2x larger KV caches. Furthermore, it leads to significant improvements in SLO attainment (up to 90-100% improvement) and TTFT (up to 71% reduction), compared to systems that do not disaggregate. The code is available at https://github.com/vbdi/epdserve. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_05460 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Efficiently Serving Large Multimodal Models Using EPD Disaggregation Singh, Gursimran Wang, Xinglu Hu, Yifan Yu, Timothy Xing, Linzi Jiang, Wei Wang, Zhefeng Bai, Xiaolong Li, Yi Xiong, Ying Zhang, Yong Fan, Zhenan Distributed, Parallel, and Cluster Computing Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Large Multimodal Models (LMMs) extend Large Language Models (LLMs) by handling diverse inputs such as images, audio, and video, but at the cost of adding a multimodal encoding stage that increases both computational and memory overhead. This step negatively affects key Service Level Objectives (SLOs), such as time to first token (TTFT) and time per output token (TPOT). We introduce Encode-Prefill-Decode (EPD) Disaggregation, a novel framework that separates the encoding, prefill, and decode stages onto dedicated resources. Unlike current systems, which bundle encoding and prefill together, our approach decouples these steps, unlocking new opportunities and optimizations. These include a mechanism to cache multimedia tokens for efficient transfer, a novel way to parallelize the encoding load within a request, a module for optimal resource allocation for disaggregated serving, and a novel role-switching method to handle changing workload characteristics. Experimental evaluations with popular LMMs show substantial gains in memory efficiency (up to 15x lower peak memory utilization), batch sizes (up to 22x larger), 10x more images per request, and 2.2x larger KV caches. Furthermore, it leads to significant improvements in SLO attainment (up to 90-100% improvement) and TTFT (up to 71% reduction), compared to systems that do not disaggregate. The code is available at https://github.com/vbdi/epdserve. |
| title | Efficiently Serving Large Multimodal Models Using EPD Disaggregation |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2501.05460 |