VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912666804551680 |
|---|---|
| author | Fu, Chaoyou Lin, Haojia Wang, Xiong Zhang, Yi-Fan Shen, Yunhang Liu, Xiaoyu Cao, Haoyu Long, Zuwei Gao, Heting Li, Ke Ma, Long Zheng, Xiawu Ji, Rongrong Sun, Xing Shan, Caifeng He, Ran |
| author_facet | Fu, Chaoyou Lin, Haojia Wang, Xiong Zhang, Yi-Fan Shen, Yunhang Liu, Xiaoyu Cao, Haoyu Long, Zuwei Gao, Heting Li, Ke Ma, Long Zheng, Xiawu Ji, Rongrong Sun, Xing Shan, Caifeng He, Ran |
| contents | Recent Multimodal Large Language Models (MLLMs) have typically focused on integrating visual and textual modalities, with less emphasis placed on the role of speech in enhancing interaction. However, speech plays a crucial role in multimodal dialogue systems, and implementing high-performance in both vision and speech tasks remains a significant challenge due to the fundamental modality differences. In this paper, we propose a carefully designed multi-stage training methodology that progressively trains LLM to understand both visual and speech information, ultimately enabling fluent vision and speech interaction. Our approach not only preserves strong vision-language capacity, but also enables efficient speech-to-speech dialogue capabilities without separate ASR and TTS modules, significantly accelerating multimodal end-to-end response speed. By comparing our method against state-of-the-art counterparts across benchmarks for image, video, and speech tasks, we demonstrate that our model is equipped with both strong visual and speech capabilities, making near real-time vision and speech interaction. Code has been released at https://github.com/VITA-MLLM/VITA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_01957 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction Fu, Chaoyou Lin, Haojia Wang, Xiong Zhang, Yi-Fan Shen, Yunhang Liu, Xiaoyu Cao, Haoyu Long, Zuwei Gao, Heting Li, Ke Ma, Long Zheng, Xiawu Ji, Rongrong Sun, Xing Shan, Caifeng He, Ran Computer Vision and Pattern Recognition Sound Audio and Speech Processing Recent Multimodal Large Language Models (MLLMs) have typically focused on integrating visual and textual modalities, with less emphasis placed on the role of speech in enhancing interaction. However, speech plays a crucial role in multimodal dialogue systems, and implementing high-performance in both vision and speech tasks remains a significant challenge due to the fundamental modality differences. In this paper, we propose a carefully designed multi-stage training methodology that progressively trains LLM to understand both visual and speech information, ultimately enabling fluent vision and speech interaction. Our approach not only preserves strong vision-language capacity, but also enables efficient speech-to-speech dialogue capabilities without separate ASR and TTS modules, significantly accelerating multimodal end-to-end response speed. By comparing our method against state-of-the-art counterparts across benchmarks for image, video, and speech tasks, we demonstrate that our model is equipped with both strong visual and speech capabilities, making near real-time vision and speech interaction. Code has been released at https://github.com/VITA-MLLM/VITA. |
| title | VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction |
| topic | Computer Vision and Pattern Recognition Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2501.01957 |