EMID: An Emotional Aligned Dataset in Audio-Visual Modality
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916350652317696 |
|---|---|
| author | Zou, Jialing Mei, Jiahao Ye, Guangze Huai, Tianyu Shen, Qiwei Dong, Daoguo |
| author_facet | Zou, Jialing Mei, Jiahao Ye, Guangze Huai, Tianyu Shen, Qiwei Dong, Daoguo |
| contents | In this paper, we propose Emotionally paired Music and Image Dataset (EMID), a novel dataset designed for the emotional matching of music and images, to facilitate auditory-visual cross-modal tasks such as generation and retrieval. Unlike existing approaches that primarily focus on semantic correlations or roughly divided emotional relations, EMID emphasizes the significance of emotional consistency between music and images using an advanced 13-dimension emotional model. By incorporating emotional alignment into the dataset, it aims to establish pairs that closely align with human perceptual understanding, thereby raising the performance of auditory-visual cross-modal tasks. We also design a supplemental module named EMI-Adapter to optimize existing cross-modal alignment methods. To validate the effectiveness of the EMID, we conduct a psychological experiment, which has demonstrated that considering the emotional relationship between the two modalities effectively improves the accuracy of matching in abstract perspective. This research lays the foundation for future cross-modal research in domains such as psychotherapy and contributes to advancing the understanding and utilization of emotions in cross-modal alignment. The EMID dataset is available at https://github.com/ecnu-aigc/EMID. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2308_07622 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | EMID: An Emotional Aligned Dataset in Audio-Visual Modality Zou, Jialing Mei, Jiahao Ye, Guangze Huai, Tianyu Shen, Qiwei Dong, Daoguo Multimedia In this paper, we propose Emotionally paired Music and Image Dataset (EMID), a novel dataset designed for the emotional matching of music and images, to facilitate auditory-visual cross-modal tasks such as generation and retrieval. Unlike existing approaches that primarily focus on semantic correlations or roughly divided emotional relations, EMID emphasizes the significance of emotional consistency between music and images using an advanced 13-dimension emotional model. By incorporating emotional alignment into the dataset, it aims to establish pairs that closely align with human perceptual understanding, thereby raising the performance of auditory-visual cross-modal tasks. We also design a supplemental module named EMI-Adapter to optimize existing cross-modal alignment methods. To validate the effectiveness of the EMID, we conduct a psychological experiment, which has demonstrated that considering the emotional relationship between the two modalities effectively improves the accuracy of matching in abstract perspective. This research lays the foundation for future cross-modal research in domains such as psychotherapy and contributes to advancing the understanding and utilization of emotions in cross-modal alignment. The EMID dataset is available at https://github.com/ecnu-aigc/EMID. |
| title | EMID: An Emotional Aligned Dataset in Audio-Visual Modality |
| topic | Multimedia |
| url | https://arxiv.org/abs/2308.07622 |