Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.00915 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910902224158720 |
|---|---|
| author | Mi, Liang Wang, Weijun Tu, Wenming He, Qingfeng Kong, Rui Fang, Xinyu Dong, Yazhu Zhang, Yikang Li, Yunchun Li, Meng Dai, Haipeng Chen, Guihai Liu, Yunxin |
| author_facet | Mi, Liang Wang, Weijun Tu, Wenming He, Qingfeng Kong, Rui Fang, Xinyu Dong, Yazhu Zhang, Yikang Li, Yunchun Li, Meng Dai, Haipeng Chen, Guihai Liu, Yunxin |
| contents | Large Multimodal Models (LMMs) have shown significant progress in various complex vision tasks with the solid linguistic and reasoning capacity inherited from large language models (LMMs). Low-rank adaptation (LoRA) offers a promising method to integrate external knowledge into LMMs, compensating for their limitations on domain-specific tasks. However, the existing LoRA model serving is excessively computationally expensive and causes extremely high latency. In this paper, we present an end-to-end solution that empowers diverse vision tasks and enriches vision applications with LoRA LMMs. Our system, VaLoRA, enables accurate and efficient vision tasks by 1) an accuracy-aware LoRA adapter generation approach that generates LoRA adapters rich in domain-specific knowledge to meet application-specific accuracy requirements, 2) an adaptive-tiling LoRA adapters batching operator that efficiently computes concurrent heterogeneous LoRA adapters, and 3) a flexible LoRA adapter orchestration mechanism that manages application requests and LoRA adapters to achieve the lowest average response latency. We prototype VaLoRA on five popular vision tasks on three LMMs. Experiment results reveal that VaLoRA improves 24-62% of the accuracy compared to the original LMMs and reduces 20-89% of the latency compared to the state-of-the-art LoRA model serving systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_00915 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Empower Vision Applications with LoRA LMM Mi, Liang Wang, Weijun Tu, Wenming He, Qingfeng Kong, Rui Fang, Xinyu Dong, Yazhu Zhang, Yikang Li, Yunchun Li, Meng Dai, Haipeng Chen, Guihai Liu, Yunxin Computer Vision and Pattern Recognition Artificial Intelligence Large Multimodal Models (LMMs) have shown significant progress in various complex vision tasks with the solid linguistic and reasoning capacity inherited from large language models (LMMs). Low-rank adaptation (LoRA) offers a promising method to integrate external knowledge into LMMs, compensating for their limitations on domain-specific tasks. However, the existing LoRA model serving is excessively computationally expensive and causes extremely high latency. In this paper, we present an end-to-end solution that empowers diverse vision tasks and enriches vision applications with LoRA LMMs. Our system, VaLoRA, enables accurate and efficient vision tasks by 1) an accuracy-aware LoRA adapter generation approach that generates LoRA adapters rich in domain-specific knowledge to meet application-specific accuracy requirements, 2) an adaptive-tiling LoRA adapters batching operator that efficiently computes concurrent heterogeneous LoRA adapters, and 3) a flexible LoRA adapter orchestration mechanism that manages application requests and LoRA adapters to achieve the lowest average response latency. We prototype VaLoRA on five popular vision tasks on three LMMs. Experiment results reveal that VaLoRA improves 24-62% of the accuracy compared to the original LMMs and reduces 20-89% of the latency compared to the state-of-the-art LoRA model serving systems. |
| title | Empower Vision Applications with LoRA LMM |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2411.00915 |