Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.18042 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908275730022400 |
|---|---|
| author | Chen, Kai Gou, Yunhao Huang, Runhui Liu, Zhili Tan, Daxin Xu, Jing Wang, Chunwei Zhu, Yi Zeng, Yihan Yang, Kuo Wang, Dingdong Xiang, Kun Li, Haoyuan Bai, Haoli Han, Jianhua Li, Xiaohui Jin, Weike Xie, Nian Zhang, Yu Kwok, James T. Zhao, Hengshuang Liang, Xiaodan Yeung, Dit-Yan Chen, Xiao Li, Zhenguo Zhang, Wei Liu, Qun Yao, Jun Hong, Lanqing Hou, Lu Xu, Hang |
| author_facet | Chen, Kai Gou, Yunhao Huang, Runhui Liu, Zhili Tan, Daxin Xu, Jing Wang, Chunwei Zhu, Yi Zeng, Yihan Yang, Kuo Wang, Dingdong Xiang, Kun Li, Haoyuan Bai, Haoli Han, Jianhua Li, Xiaohui Jin, Weike Xie, Nian Zhang, Yu Kwok, James T. Zhao, Hengshuang Liang, Xiaodan Yeung, Dit-Yan Chen, Xiao Li, Zhenguo Zhang, Wei Liu, Qun Yao, Jun Hong, Lanqing Hou, Lu Xu, Hang |
| contents | GPT-4o, an omni-modal model that enables vocal conversations with diverse emotions and tones, marks a milestone for omni-modal foundation models. However, empowering Large Language Models to perceive and generate images, texts, and speeches end-to-end with publicly available data remains challenging for the open-source community. Existing vision-language models rely on external tools for speech processing, while speech-language models still suffer from limited or totally without vision-understanding capabilities. To address this gap, we propose the EMOVA (EMotionally Omni-present Voice Assistant), to enable Large Language Models with end-to-end speech abilities while maintaining the leading vision-language performance. With a semantic-acoustic disentangled speech tokenizer, we surprisingly notice that omni-modal alignment can further enhance vision-language and speech abilities compared with the bi-modal aligned counterparts. Moreover, a lightweight style module is introduced for the flexible speech style controls including emotions and pitches. For the first time, EMOVA achieves state-of-the-art performance on both the vision-language and speech benchmarks, and meanwhile, supporting omni-modal spoken dialogue with vivid emotions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_18042 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions Chen, Kai Gou, Yunhao Huang, Runhui Liu, Zhili Tan, Daxin Xu, Jing Wang, Chunwei Zhu, Yi Zeng, Yihan Yang, Kuo Wang, Dingdong Xiang, Kun Li, Haoyuan Bai, Haoli Han, Jianhua Li, Xiaohui Jin, Weike Xie, Nian Zhang, Yu Kwok, James T. Zhao, Hengshuang Liang, Xiaodan Yeung, Dit-Yan Chen, Xiao Li, Zhenguo Zhang, Wei Liu, Qun Yao, Jun Hong, Lanqing Hou, Lu Xu, Hang Computer Vision and Pattern Recognition Computation and Language GPT-4o, an omni-modal model that enables vocal conversations with diverse emotions and tones, marks a milestone for omni-modal foundation models. However, empowering Large Language Models to perceive and generate images, texts, and speeches end-to-end with publicly available data remains challenging for the open-source community. Existing vision-language models rely on external tools for speech processing, while speech-language models still suffer from limited or totally without vision-understanding capabilities. To address this gap, we propose the EMOVA (EMotionally Omni-present Voice Assistant), to enable Large Language Models with end-to-end speech abilities while maintaining the leading vision-language performance. With a semantic-acoustic disentangled speech tokenizer, we surprisingly notice that omni-modal alignment can further enhance vision-language and speech abilities compared with the bi-modal aligned counterparts. Moreover, a lightweight style module is introduced for the flexible speech style controls including emotions and pitches. For the first time, EMOVA achieves state-of-the-art performance on both the vision-language and speech benchmarks, and meanwhile, supporting omni-modal spoken dialogue with vivid emotions. |
| title | EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2409.18042 |