Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.14177 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914877182836736 |
|---|---|
| author | Chen, Kaibing Shen, Dong Zhong, Hanwen Zhong, Huasong Xia, Kui Xu, Di Yuan, Wei Hu, Yifei Wen, Bin Zhang, Tianke Liu, Changyi Fan, Dewen Xiao, Huihui Wu, Jiahong Yang, Fan Li, Size Zhang, Di |
| author_facet | Chen, Kaibing Shen, Dong Zhong, Hanwen Zhong, Huasong Xia, Kui Xu, Di Yuan, Wei Hu, Yifei Wen, Bin Zhang, Tianke Liu, Changyi Fan, Dewen Xiao, Huihui Wu, Jiahong Yang, Fan Li, Size Zhang, Di |
| contents | In the field of multi-modal language models, the majority of methods are built on an architecture similar to LLaVA. These models use a single-layer ViT feature as a visual prompt, directly feeding it into the language models alongside textual tokens. However, when dealing with long sequences of visual signals or inputs such as videos, the self-attention mechanism of language models can lead to significant computational overhead. Additionally, using single-layer ViT features makes it challenging for large language models to perceive visual signals fully. This paper proposes an efficient multi-modal language model to minimize computational costs while enabling the model to perceive visual signals as comprehensively as possible. Our method primarily includes: (1) employing cross-attention to image-text interaction similar to Flamingo. (2) utilize hierarchical ViT features. (3) introduce the Mixture of Experts (MoE) mechanism to enhance model effectiveness. Our model achieves competitive scores on public multi-modal benchmarks and performs well in tasks such as image captioning and video captioning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_14177 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | EVLM: An Efficient Vision-Language Model for Visual Understanding Chen, Kaibing Shen, Dong Zhong, Hanwen Zhong, Huasong Xia, Kui Xu, Di Yuan, Wei Hu, Yifei Wen, Bin Zhang, Tianke Liu, Changyi Fan, Dewen Xiao, Huihui Wu, Jiahong Yang, Fan Li, Size Zhang, Di Computer Vision and Pattern Recognition In the field of multi-modal language models, the majority of methods are built on an architecture similar to LLaVA. These models use a single-layer ViT feature as a visual prompt, directly feeding it into the language models alongside textual tokens. However, when dealing with long sequences of visual signals or inputs such as videos, the self-attention mechanism of language models can lead to significant computational overhead. Additionally, using single-layer ViT features makes it challenging for large language models to perceive visual signals fully. This paper proposes an efficient multi-modal language model to minimize computational costs while enabling the model to perceive visual signals as comprehensively as possible. Our method primarily includes: (1) employing cross-attention to image-text interaction similar to Flamingo. (2) utilize hierarchical ViT features. (3) introduce the Mixture of Experts (MoE) mechanism to enhance model effectiveness. Our model achieves competitive scores on public multi-modal benchmarks and performs well in tasks such as image captioning and video captioning. |
| title | EVLM: An Efficient Vision-Language Model for Visual Understanding |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.14177 |