Variational Multi-Modal Hypergraph Attention Network for Multi-Modal Relation Extraction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909173981118464 |
|---|---|
| author | Li, Qian Ji, Cheng Guo, Shu Zhao, Yong Mao, Qianren Wang, Shangguang Wei, Yuntao Li, Jianxin |
| author_facet | Li, Qian Ji, Cheng Guo, Shu Zhao, Yong Mao, Qianren Wang, Shangguang Wei, Yuntao Li, Jianxin |
| contents | Multi-modal relation extraction (MMRE) is a challenging task that aims to identify relations between entities in text leveraging image information. Existing methods are limited by their neglect of the multiple entity pairs in one sentence sharing very similar contextual information (ie, the same text and image), resulting in increased difficulty in the MMRE task. To address this limitation, we propose the Variational Multi-Modal Hypergraph Attention Network (VM-HAN) for multi-modal relation extraction. Specifically, we first construct a multi-modal hypergraph for each sentence with the corresponding image, to establish different high-order intra-/inter-modal correlations for different entity pairs in each sentence. We further design the Variational Hypergraph Attention Networks (V-HAN) to obtain representational diversity among different entity pairs using Gaussian distribution and learn a better hypergraph structure via variational attention. VM-HAN achieves state-of-the-art performance on the multi-modal relation extraction task, outperforming existing methods in terms of accuracy and efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_12006 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Variational Multi-Modal Hypergraph Attention Network for Multi-Modal Relation Extraction Li, Qian Ji, Cheng Guo, Shu Zhao, Yong Mao, Qianren Wang, Shangguang Wei, Yuntao Li, Jianxin Computation and Language Multi-modal relation extraction (MMRE) is a challenging task that aims to identify relations between entities in text leveraging image information. Existing methods are limited by their neglect of the multiple entity pairs in one sentence sharing very similar contextual information (ie, the same text and image), resulting in increased difficulty in the MMRE task. To address this limitation, we propose the Variational Multi-Modal Hypergraph Attention Network (VM-HAN) for multi-modal relation extraction. Specifically, we first construct a multi-modal hypergraph for each sentence with the corresponding image, to establish different high-order intra-/inter-modal correlations for different entity pairs in each sentence. We further design the Variational Hypergraph Attention Networks (V-HAN) to obtain representational diversity among different entity pairs using Gaussian distribution and learn a better hypergraph structure via variational attention. VM-HAN achieves state-of-the-art performance on the multi-modal relation extraction task, outperforming existing methods in terms of accuracy and efficiency. |
| title | Variational Multi-Modal Hypergraph Attention Network for Multi-Modal Relation Extraction |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2404.12006 |