DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910743975165952 |
|---|---|
| author | Wu, Zhiyu Chen, Xiaokang Pan, Zizheng Liu, Xingchao Liu, Wen Dai, Damai Gao, Huazuo Ma, Yiyang Wu, Chengyue Wang, Bingxuan Xie, Zhenda Wu, Yu Hu, Kai Wang, Jiawei Sun, Yaofeng Li, Yukun Piao, Yishi Guan, Kang Liu, Aixin Xie, Xin You, Yuxiang Dong, Kai Yu, Xingkai Zhang, Haowei Zhao, Liang Wang, Yisong Ruan, Chong |
| author_facet | Wu, Zhiyu Chen, Xiaokang Pan, Zizheng Liu, Xingchao Liu, Wen Dai, Damai Gao, Huazuo Ma, Yiyang Wu, Chengyue Wang, Bingxuan Xie, Zhenda Wu, Yu Hu, Kai Wang, Jiawei Sun, Yaofeng Li, Yukun Piao, Yishi Guan, Kang Liu, Aixin Xie, Xin You, Yuxiang Dong, Kai Yu, Xingkai Zhang, Haowei Zhao, Liang Wang, Yisong Ruan, Chong |
| contents | We present DeepSeek-VL2, an advanced series of large Mixture-of-Experts (MoE) Vision-Language Models that significantly improves upon its predecessor, DeepSeek-VL, through two key major upgrades. For the vision component, we incorporate a dynamic tiling vision encoding strategy designed for processing high-resolution images with different aspect ratios. For the language component, we leverage DeepSeekMoE models with the Multi-head Latent Attention mechanism, which compresses Key-Value cache into latent vectors, to enable efficient inference and high throughput. Trained on an improved vision-language dataset, DeepSeek-VL2 demonstrates superior capabilities across various tasks, including but not limited to visual question answering, optical character recognition, document/table/chart understanding, and visual grounding. Our model series is composed of three variants: DeepSeek-VL2-Tiny, DeepSeek-VL2-Small and DeepSeek-VL2, with 1.0B, 2.8B and 4.5B activated parameters respectively. DeepSeek-VL2 achieves competitive or state-of-the-art performance with similar or fewer activated parameters compared to existing open-source dense and MoE-based models. Codes and pre-trained models are publicly accessible at https://github.com/deepseek-ai/DeepSeek-VL2. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_10302 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding Wu, Zhiyu Chen, Xiaokang Pan, Zizheng Liu, Xingchao Liu, Wen Dai, Damai Gao, Huazuo Ma, Yiyang Wu, Chengyue Wang, Bingxuan Xie, Zhenda Wu, Yu Hu, Kai Wang, Jiawei Sun, Yaofeng Li, Yukun Piao, Yishi Guan, Kang Liu, Aixin Xie, Xin You, Yuxiang Dong, Kai Yu, Xingkai Zhang, Haowei Zhao, Liang Wang, Yisong Ruan, Chong Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language We present DeepSeek-VL2, an advanced series of large Mixture-of-Experts (MoE) Vision-Language Models that significantly improves upon its predecessor, DeepSeek-VL, through two key major upgrades. For the vision component, we incorporate a dynamic tiling vision encoding strategy designed for processing high-resolution images with different aspect ratios. For the language component, we leverage DeepSeekMoE models with the Multi-head Latent Attention mechanism, which compresses Key-Value cache into latent vectors, to enable efficient inference and high throughput. Trained on an improved vision-language dataset, DeepSeek-VL2 demonstrates superior capabilities across various tasks, including but not limited to visual question answering, optical character recognition, document/table/chart understanding, and visual grounding. Our model series is composed of three variants: DeepSeek-VL2-Tiny, DeepSeek-VL2-Small and DeepSeek-VL2, with 1.0B, 2.8B and 4.5B activated parameters respectively. DeepSeek-VL2 achieves competitive or state-of-the-art performance with similar or fewer activated parameters compared to existing open-source dense and MoE-based models. Codes and pre-trained models are publicly accessible at https://github.com/deepseek-ai/DeepSeek-VL2. |
| title | DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2412.10302 |