Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.04908 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912053686435840 |
|---|---|
| author | Cai, Kaiwen Duan, Zhekai Liu, Gaowen Fleming, Charles Lu, Chris Xiaoxuan |
| author_facet | Cai, Kaiwen Duan, Zhekai Liu, Gaowen Fleming, Charles Lu, Chris Xiaoxuan |
| contents | Recent advancements in Vision-Language (VL) models have sparked interest in their deployment on edge devices, yet challenges in handling diverse visual modalities, manual annotation, and computational constraints remain. We introduce EdgeVL, a novel framework that bridges this gap by seamlessly integrating dual-modality knowledge distillation and quantization-aware contrastive learning. This approach enables the adaptation of large VL models, like CLIP, for efficient use with both RGB and non-RGB images on resource-limited devices without the need for manual annotations. EdgeVL not only transfers visual language alignment capabilities to compact models but also maintains feature quality post-quantization, significantly enhancing open-vocabulary classification performance across various visual modalities. Our work represents the first systematic effort to adapt large VL models for edge deployment, showcasing up to 15.4% accuracy improvements on multiple datasets and up to 93-fold reduction in model size. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_04908 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Self-Adapting Large Visual-Language Models to Edge Devices across Visual Modalities Cai, Kaiwen Duan, Zhekai Liu, Gaowen Fleming, Charles Lu, Chris Xiaoxuan Computer Vision and Pattern Recognition Recent advancements in Vision-Language (VL) models have sparked interest in their deployment on edge devices, yet challenges in handling diverse visual modalities, manual annotation, and computational constraints remain. We introduce EdgeVL, a novel framework that bridges this gap by seamlessly integrating dual-modality knowledge distillation and quantization-aware contrastive learning. This approach enables the adaptation of large VL models, like CLIP, for efficient use with both RGB and non-RGB images on resource-limited devices without the need for manual annotations. EdgeVL not only transfers visual language alignment capabilities to compact models but also maintains feature quality post-quantization, significantly enhancing open-vocabulary classification performance across various visual modalities. Our work represents the first systematic effort to adapt large VL models for edge deployment, showcasing up to 15.4% accuracy improvements on multiple datasets and up to 93-fold reduction in model size. |
| title | Self-Adapting Large Visual-Language Models to Edge Devices across Visual Modalities |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.04908 |