MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912002010513408 |
|---|---|
| author | Jiang, Chaoya Hongrui, Jia Xu, Haiyang Ye, Wei Dong, Mengfan Yan, Ming Zhang, Ji Huang, Fei Zhang, Shikun |
| author_facet | Jiang, Chaoya Hongrui, Jia Xu, Haiyang Ye, Wei Dong, Mengfan Yan, Ming Zhang, Ji Huang, Fei Zhang, Shikun |
| contents | This paper presents MaVEn, an innovative Multi-granularity Visual Encoding framework designed to enhance the capabilities of Multimodal Large Language Models (MLLMs) in multi-image reasoning. Current MLLMs primarily focus on single-image visual understanding, limiting their ability to interpret and integrate information across multiple images. MaVEn addresses this limitation by combining discrete visual symbol sequences, which abstract coarse-grained semantic concepts, with traditional continuous representation sequences that model fine-grained features. This dual approach bridges the semantic gap between visual and textual data, thereby improving the model's ability to process and interpret information from multiple images effectively. Additionally, we design a dynamic reduction mechanism by for long-sequence continuous features to enhance multi-image processing efficiency. Experimental results demonstrate that MaVEn significantly enhances MLLMs' understanding in complex multi-image scenarios, while also improving performance in single-image contexts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_12321 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model Jiang, Chaoya Hongrui, Jia Xu, Haiyang Ye, Wei Dong, Mengfan Yan, Ming Zhang, Ji Huang, Fei Zhang, Shikun Computation and Language Computer Vision and Pattern Recognition Multimedia This paper presents MaVEn, an innovative Multi-granularity Visual Encoding framework designed to enhance the capabilities of Multimodal Large Language Models (MLLMs) in multi-image reasoning. Current MLLMs primarily focus on single-image visual understanding, limiting their ability to interpret and integrate information across multiple images. MaVEn addresses this limitation by combining discrete visual symbol sequences, which abstract coarse-grained semantic concepts, with traditional continuous representation sequences that model fine-grained features. This dual approach bridges the semantic gap between visual and textual data, thereby improving the model's ability to process and interpret information from multiple images effectively. Additionally, we design a dynamic reduction mechanism by for long-sequence continuous features to enhance multi-image processing efficiency. Experimental results demonstrate that MaVEn significantly enhances MLLMs' understanding in complex multi-image scenarios, while also improving performance in single-image contexts. |
| title | MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model |
| topic | Computation and Language Computer Vision and Pattern Recognition Multimedia |
| url | https://arxiv.org/abs/2408.12321 |