VITA: Towards Open-Source Interactive Omni Multimodal LLM
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913867003592704 |
|---|---|
| author | Fu, Chaoyou Lin, Haojia Long, Zuwei Shen, Yunhang Dai, Yuhang Zhao, Meng Zhang, Yi-Fan Dong, Shaoqi Li, Yangze Wang, Xiong Cao, Haoyu Yin, Di Ma, Long Zheng, Xiawu Ji, Rongrong Wu, Yunsheng He, Ran Shan, Caifeng Sun, Xing |
| author_facet | Fu, Chaoyou Lin, Haojia Long, Zuwei Shen, Yunhang Dai, Yuhang Zhao, Meng Zhang, Yi-Fan Dong, Shaoqi Li, Yangze Wang, Xiong Cao, Haoyu Yin, Di Ma, Long Zheng, Xiawu Ji, Rongrong Wu, Yunsheng He, Ran Shan, Caifeng Sun, Xing |
| contents | The remarkable multimodal capabilities and interactive experience of GPT-4o underscore their necessity in practical applications, yet open-source models rarely excel in both areas. In this paper, we introduce VITA, the first-ever open-source Multimodal Large Language Model (MLLM) adept at simultaneous processing and analysis of Video, Image, Text, and Audio modalities, and meanwhile has an advanced multimodal interactive experience. Starting from Mixtral 8x7B as a language foundation, we expand its Chinese vocabulary followed by bilingual instruction tuning. We further endow the language model with visual and audio capabilities through two-stage multi-task learning of multimodal alignment and instruction tuning. VITA demonstrates robust foundational capabilities of multilingual, vision, and audio understanding, as evidenced by its strong performance across a range of both unimodal and multimodal benchmarks. Beyond foundational capabilities, we have made considerable progress in enhancing the natural multimodal human-computer interaction experience. VITA is the first step for the open-source community to explore the seamless integration of multimodal understanding and interaction. While there is still lots of work to be done on VITA to get close to close-source counterparts, we hope that its role as a pioneer can serve as a cornerstone for subsequent research. Project Page: https://vita-home.github.io. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_05211 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | VITA: Towards Open-Source Interactive Omni Multimodal LLM Fu, Chaoyou Lin, Haojia Long, Zuwei Shen, Yunhang Dai, Yuhang Zhao, Meng Zhang, Yi-Fan Dong, Shaoqi Li, Yangze Wang, Xiong Cao, Haoyu Yin, Di Ma, Long Zheng, Xiawu Ji, Rongrong Wu, Yunsheng He, Ran Shan, Caifeng Sun, Xing Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language The remarkable multimodal capabilities and interactive experience of GPT-4o underscore their necessity in practical applications, yet open-source models rarely excel in both areas. In this paper, we introduce VITA, the first-ever open-source Multimodal Large Language Model (MLLM) adept at simultaneous processing and analysis of Video, Image, Text, and Audio modalities, and meanwhile has an advanced multimodal interactive experience. Starting from Mixtral 8x7B as a language foundation, we expand its Chinese vocabulary followed by bilingual instruction tuning. We further endow the language model with visual and audio capabilities through two-stage multi-task learning of multimodal alignment and instruction tuning. VITA demonstrates robust foundational capabilities of multilingual, vision, and audio understanding, as evidenced by its strong performance across a range of both unimodal and multimodal benchmarks. Beyond foundational capabilities, we have made considerable progress in enhancing the natural multimodal human-computer interaction experience. VITA is the first step for the open-source community to explore the seamless integration of multimodal understanding and interaction. While there is still lots of work to be done on VITA to get close to close-source counterparts, we hope that its role as a pioneer can serve as a cornerstone for subsequent research. Project Page: https://vita-home.github.io. |
| title | VITA: Towards Open-Source Interactive Omni Multimodal LLM |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2408.05211 |