Emotion-Qwen: A Unified Framework for Emotion and Vision Understanding
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912535175757824 |
|---|---|
| author | Huang, Dawei Li, Qing Yan, Chuan Cheng, Zebang Han, Zihao Huang, Yurong Li, Xiang Li, Bin Wang, Xiaohui Lian, Zheng Cheng, Zhi-Qi Peng, Xiaojiang |
| author_facet | Huang, Dawei Li, Qing Yan, Chuan Cheng, Zebang Han, Zihao Huang, Yurong Li, Xiang Li, Bin Wang, Xiaohui Lian, Zheng Cheng, Zhi-Qi Peng, Xiaojiang |
| contents | Accurate emotion understanding in videos necessitates effectively recognizing and interpreting emotional states by integrating visual, textual, auditory, and contextual cues. Although recent Large Multimodal Models (LMMs) have exhibited significant progress in general vision-language (VL) tasks, their performance often deteriorates in emotion-specific scenarios, exhibiting catastrophic forgetting when fine-tuned on emotion-centric tasks. To overcome these limitations, we propose Emotion-Qwen, a unified multimodal framework designed to simultaneously enable robust emotion understanding and preserve general VL reasoning capabilities. Emotion-Qwen introduces a novel Hybrid Compressor based on a Mixture-of-Experts (MoE) architecture, dynamically routing inputs to optimally balance emotion-specific processing and general multimodal reasoning. We further propose a carefully structured three-stage pre-training pipeline, leveraging extensive general and emotion-focused datasets to strengthen multimodal representation robustness and model adaptability. Additionally, we develop the Video Emotion Reasoning (VER) dataset, a large-scale bilingual resource containing over 40K video clips annotated with detailed context-aware emotional descriptions, significantly facilitating research on fine-grained emotional reasoning. Extensive experiments confirm that Emotion-Qwen achieves state-of-the-art performance across multiple emotion recognition and reasoning benchmarks, while maintaining highly competitive results in general VL tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_06685 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Emotion-Qwen: A Unified Framework for Emotion and Vision Understanding Huang, Dawei Li, Qing Yan, Chuan Cheng, Zebang Han, Zihao Huang, Yurong Li, Xiang Li, Bin Wang, Xiaohui Lian, Zheng Cheng, Zhi-Qi Peng, Xiaojiang Multimedia Computer Vision and Pattern Recognition Accurate emotion understanding in videos necessitates effectively recognizing and interpreting emotional states by integrating visual, textual, auditory, and contextual cues. Although recent Large Multimodal Models (LMMs) have exhibited significant progress in general vision-language (VL) tasks, their performance often deteriorates in emotion-specific scenarios, exhibiting catastrophic forgetting when fine-tuned on emotion-centric tasks. To overcome these limitations, we propose Emotion-Qwen, a unified multimodal framework designed to simultaneously enable robust emotion understanding and preserve general VL reasoning capabilities. Emotion-Qwen introduces a novel Hybrid Compressor based on a Mixture-of-Experts (MoE) architecture, dynamically routing inputs to optimally balance emotion-specific processing and general multimodal reasoning. We further propose a carefully structured three-stage pre-training pipeline, leveraging extensive general and emotion-focused datasets to strengthen multimodal representation robustness and model adaptability. Additionally, we develop the Video Emotion Reasoning (VER) dataset, a large-scale bilingual resource containing over 40K video clips annotated with detailed context-aware emotional descriptions, significantly facilitating research on fine-grained emotional reasoning. Extensive experiments confirm that Emotion-Qwen achieves state-of-the-art performance across multiple emotion recognition and reasoning benchmarks, while maintaining highly competitive results in general VL tasks. |
| title | Emotion-Qwen: A Unified Framework for Emotion and Vision Understanding |
| topic | Multimedia Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.06685 |