Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912101551833088 |
|---|---|
| author | Cheng, Zebang Cheng, Zhi-Qi He, Jun-Yan Sun, Jingdong Wang, Kai Lin, Yuxiang Lian, Zheng Peng, Xiaojiang Hauptmann, Alexander |
| author_facet | Cheng, Zebang Cheng, Zhi-Qi He, Jun-Yan Sun, Jingdong Wang, Kai Lin, Yuxiang Lian, Zheng Peng, Xiaojiang Hauptmann, Alexander |
| contents | Accurate emotion perception is crucial for various applications, including human-computer interaction, education, and counseling. However, traditional single-modality approaches often fail to capture the complexity of real-world emotional expressions, which are inherently multimodal. Moreover, existing Multimodal Large Language Models (MLLMs) face challenges in integrating audio and recognizing subtle facial micro-expressions. To address this, we introduce the MERR dataset, containing 28,618 coarse-grained and 4,487 fine-grained annotated samples across diverse emotional categories. This dataset enables models to learn from varied scenarios and generalize to real-world applications. Furthermore, we propose Emotion-LLaMA, a model that seamlessly integrates audio, visual, and textual inputs through emotion-specific encoders. By aligning features into a shared space and employing a modified LLaMA model with instruction tuning, Emotion-LLaMA significantly enhances both emotional recognition and reasoning capabilities. Extensive evaluations show Emotion-LLaMA outperforms other MLLMs, achieving top scores in Clue Overlap (7.83) and Label Overlap (6.25) on EMER, an F1 score of 0.9036 on MER2023-SEMI challenge, and the highest UAR (45.59) and WAR (59.37) in zero-shot evaluations on DFEW dataset. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_11161 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning Cheng, Zebang Cheng, Zhi-Qi He, Jun-Yan Sun, Jingdong Wang, Kai Lin, Yuxiang Lian, Zheng Peng, Xiaojiang Hauptmann, Alexander Artificial Intelligence Multimedia Accurate emotion perception is crucial for various applications, including human-computer interaction, education, and counseling. However, traditional single-modality approaches often fail to capture the complexity of real-world emotional expressions, which are inherently multimodal. Moreover, existing Multimodal Large Language Models (MLLMs) face challenges in integrating audio and recognizing subtle facial micro-expressions. To address this, we introduce the MERR dataset, containing 28,618 coarse-grained and 4,487 fine-grained annotated samples across diverse emotional categories. This dataset enables models to learn from varied scenarios and generalize to real-world applications. Furthermore, we propose Emotion-LLaMA, a model that seamlessly integrates audio, visual, and textual inputs through emotion-specific encoders. By aligning features into a shared space and employing a modified LLaMA model with instruction tuning, Emotion-LLaMA significantly enhances both emotional recognition and reasoning capabilities. Extensive evaluations show Emotion-LLaMA outperforms other MLLMs, achieving top scores in Clue Overlap (7.83) and Label Overlap (6.25) on EMER, an F1 score of 0.9036 on MER2023-SEMI challenge, and the highest UAR (45.59) and WAR (59.37) in zero-shot evaluations on DFEW dataset. |
| title | Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning |
| topic | Artificial Intelligence Multimedia |
| url | https://arxiv.org/abs/2406.11161 |