NativE: Multi-modal Knowledge Graph Completion in the Wild
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914848737067008 |
|---|---|
| author | Zhang, Yichi Chen, Zhuo Guo, Lingbing Xu, Yajing Hu, Binbin Liu, Ziqi Zhang, Wen Chen, Huajun |
| author_facet | Zhang, Yichi Chen, Zhuo Guo, Lingbing Xu, Yajing Hu, Binbin Liu, Ziqi Zhang, Wen Chen, Huajun |
| contents | Multi-modal knowledge graph completion (MMKGC) aims to automatically discover the unobserved factual knowledge from a given multi-modal knowledge graph by collaboratively modeling the triple structure and multi-modal information from entities. However, real-world MMKGs present challenges due to their diverse and imbalanced nature, which means that the modality information can span various types (e.g., image, text, numeric, audio, video) but its distribution among entities is uneven, leading to missing modalities for certain entities. Existing works usually focus on common modalities like image and text while neglecting the imbalanced distribution phenomenon of modal information. To address these issues, we propose a comprehensive framework NativE to achieve MMKGC in the wild. NativE proposes a relation-guided dual adaptive fusion module that enables adaptive fusion for any modalities and employs a collaborative modality adversarial training framework to augment the imbalanced modality information. We construct a new benchmark called WildKGC with five datasets to evaluate our method. The empirical results compared with 21 recent baselines confirm the superiority of our method, consistently achieving state-of-the-art performance across different datasets and various scenarios while keeping efficient and generalizable. Our code and data are released at https://github.com/zjukg/NATIVE |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_17605 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | NativE: Multi-modal Knowledge Graph Completion in the Wild Zhang, Yichi Chen, Zhuo Guo, Lingbing Xu, Yajing Hu, Binbin Liu, Ziqi Zhang, Wen Chen, Huajun Multimedia Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Information Retrieval Multi-modal knowledge graph completion (MMKGC) aims to automatically discover the unobserved factual knowledge from a given multi-modal knowledge graph by collaboratively modeling the triple structure and multi-modal information from entities. However, real-world MMKGs present challenges due to their diverse and imbalanced nature, which means that the modality information can span various types (e.g., image, text, numeric, audio, video) but its distribution among entities is uneven, leading to missing modalities for certain entities. Existing works usually focus on common modalities like image and text while neglecting the imbalanced distribution phenomenon of modal information. To address these issues, we propose a comprehensive framework NativE to achieve MMKGC in the wild. NativE proposes a relation-guided dual adaptive fusion module that enables adaptive fusion for any modalities and employs a collaborative modality adversarial training framework to augment the imbalanced modality information. We construct a new benchmark called WildKGC with five datasets to evaluate our method. The empirical results compared with 21 recent baselines confirm the superiority of our method, consistently achieving state-of-the-art performance across different datasets and various scenarios while keeping efficient and generalizable. Our code and data are released at https://github.com/zjukg/NATIVE |
| title | NativE: Multi-modal Knowledge Graph Completion in the Wild |
| topic | Multimedia Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Information Retrieval |
| url | https://arxiv.org/abs/2406.17605 |