LiteVLM: A Low-Latency Vision-Language Model Inference Pipeline for Resource-Constrained Environments
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914127204581376 |
|---|---|
| author | Huang, Jin Jin, Yuchao An, Le Park, Josh |
| author_facet | Huang, Jin Jin, Yuchao An, Le Park, Josh |
| contents | This paper introduces an efficient Vision-Language Model (VLM) pipeline specifically optimized for deployment on embedded devices, such as those used in robotics and autonomous driving. The pipeline significantly reduces the computational overhead by jointly leveraging patch selection to filter irrelevant camera views, a token selection module to reduce input sequence length for the LLM, and speculative decoding to accelerate token generation. Evaluation on the NVIDIA DRIVE Thor platform for automonous driving application, our pipeline achieves $2.5\times$ end-to-end latency reduction without compromising task accuracy. The speed-up further increases to $3.2\times$ when applying FP8 post-training quantization. These results demonstrate our pipeline as a viable solution for enabling real-time VLM deployment in resource-constrained environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_07416 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LiteVLM: A Low-Latency Vision-Language Model Inference Pipeline for Resource-Constrained Environments Huang, Jin Jin, Yuchao An, Le Park, Josh Machine Learning Artificial Intelligence This paper introduces an efficient Vision-Language Model (VLM) pipeline specifically optimized for deployment on embedded devices, such as those used in robotics and autonomous driving. The pipeline significantly reduces the computational overhead by jointly leveraging patch selection to filter irrelevant camera views, a token selection module to reduce input sequence length for the LLM, and speculative decoding to accelerate token generation. Evaluation on the NVIDIA DRIVE Thor platform for automonous driving application, our pipeline achieves $2.5\times$ end-to-end latency reduction without compromising task accuracy. The speed-up further increases to $3.2\times$ when applying FP8 post-training quantization. These results demonstrate our pipeline as a viable solution for enabling real-time VLM deployment in resource-constrained environments. |
| title | LiteVLM: A Low-Latency Vision-Language Model Inference Pipeline for Resource-Constrained Environments |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2506.07416 |