Noise-powered Multi-modal Knowledge Graph Representation Framework
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913650220990464 |
|---|---|
| author | Chen, Zhuo Fang, Yin Zhang, Yichi Guo, Lingbing Chen, Jiaoyan Pan, Jeff Z. Chen, Huajun Zhang, Wen |
| author_facet | Chen, Zhuo Fang, Yin Zhang, Yichi Guo, Lingbing Chen, Jiaoyan Pan, Jeff Z. Chen, Huajun Zhang, Wen |
| contents | The rise of Multi-modal Pre-training highlights the necessity for a unified Multi-Modal Knowledge Graph (MMKG) representation learning framework. Such a framework is essential for embedding structured knowledge into multi-modal Large Language Models effectively, alleviating issues like knowledge misconceptions and multi-modal hallucinations. In this work, we explore the efficacy of models in accurately embedding entities within MMKGs through two pivotal tasks: Multi-modal Knowledge Graph Completion (MKGC) and Multi-modal Entity Alignment (MMEA). Building on this foundation, we propose a novel SNAG method that utilizes a Transformer-based architecture equipped with modality-level noise masking to robustly integrate multi-modal entity features in KGs. By incorporating specific training objectives for both MKGC and MMEA, our approach achieves SOTA performance across a total of ten datasets, demonstrating its versatility. Moreover, SNAG can not only function as a standalone model but also enhance other existing methods, providing stable performance improvements. Code and data are available at https://github.com/zjukg/SNAG. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_06832 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Noise-powered Multi-modal Knowledge Graph Representation Framework Chen, Zhuo Fang, Yin Zhang, Yichi Guo, Lingbing Chen, Jiaoyan Pan, Jeff Z. Chen, Huajun Zhang, Wen Computation and Language Artificial Intelligence The rise of Multi-modal Pre-training highlights the necessity for a unified Multi-Modal Knowledge Graph (MMKG) representation learning framework. Such a framework is essential for embedding structured knowledge into multi-modal Large Language Models effectively, alleviating issues like knowledge misconceptions and multi-modal hallucinations. In this work, we explore the efficacy of models in accurately embedding entities within MMKGs through two pivotal tasks: Multi-modal Knowledge Graph Completion (MKGC) and Multi-modal Entity Alignment (MMEA). Building on this foundation, we propose a novel SNAG method that utilizes a Transformer-based architecture equipped with modality-level noise masking to robustly integrate multi-modal entity features in KGs. By incorporating specific training objectives for both MKGC and MMEA, our approach achieves SOTA performance across a total of ten datasets, demonstrating its versatility. Moreover, SNAG can not only function as a standalone model but also enhance other existing methods, providing stable performance improvements. Code and data are available at https://github.com/zjukg/SNAG. |
| title | Noise-powered Multi-modal Knowledge Graph Representation Framework |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2403.06832 |