Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.20339 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911894874357760 |
|---|---|
| author | Ma, Feipeng Xue, Hongwei Wang, Guangting Zhou, Yizhou Rao, Fengyun Yan, Shilin Zhang, Yueyi Wu, Siying Shou, Mike Zheng Sun, Xiaoyan |
| author_facet | Ma, Feipeng Xue, Hongwei Wang, Guangting Zhou, Yizhou Rao, Fengyun Yan, Shilin Zhang, Yueyi Wu, Siying Shou, Mike Zheng Sun, Xiaoyan |
| contents | Existing Multimodal Large Language Models (MLLMs) follow the paradigm that perceives visual information by aligning visual features with the input space of Large Language Models (LLMs), and concatenating visual tokens with text tokens to form a unified sequence input for LLMs. These methods demonstrate promising results on various vision-language tasks but are limited by the high computational effort due to the extended input sequence resulting from the involvement of visual tokens. In this paper, instead of input space alignment, we propose a novel parameter space alignment paradigm that represents visual information as model weights. For each input image, we use a vision encoder to extract visual features, convert features into perceptual weights, and merge the perceptual weights with LLM's weights. In this way, the input of LLM does not require visual tokens, which reduces the length of the input sequence and greatly improves efficiency. Following this paradigm, we propose VLoRA with the perceptual weights generator. The perceptual weights generator is designed to convert visual features to perceptual weights with low-rank property, exhibiting a form similar to LoRA. The experimental results show that our VLoRA achieves comparable performance on various benchmarks for MLLMs, while significantly reducing the computational costs for both training and inference. The code and models will be made open-source. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_20339 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Visual Perception by Large Language Model's Weights Ma, Feipeng Xue, Hongwei Wang, Guangting Zhou, Yizhou Rao, Fengyun Yan, Shilin Zhang, Yueyi Wu, Siying Shou, Mike Zheng Sun, Xiaoyan Computer Vision and Pattern Recognition Existing Multimodal Large Language Models (MLLMs) follow the paradigm that perceives visual information by aligning visual features with the input space of Large Language Models (LLMs), and concatenating visual tokens with text tokens to form a unified sequence input for LLMs. These methods demonstrate promising results on various vision-language tasks but are limited by the high computational effort due to the extended input sequence resulting from the involvement of visual tokens. In this paper, instead of input space alignment, we propose a novel parameter space alignment paradigm that represents visual information as model weights. For each input image, we use a vision encoder to extract visual features, convert features into perceptual weights, and merge the perceptual weights with LLM's weights. In this way, the input of LLM does not require visual tokens, which reduces the length of the input sequence and greatly improves efficiency. Following this paradigm, we propose VLoRA with the perceptual weights generator. The perceptual weights generator is designed to convert visual features to perceptual weights with low-rank property, exhibiting a form similar to LoRA. The experimental results show that our VLoRA achieves comparable performance on various benchmarks for MLLMs, while significantly reducing the computational costs for both training and inference. The code and models will be made open-source. |
| title | Visual Perception by Large Language Model's Weights |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2405.20339 |