GT2Vec: Large Language Models as Multi-Modal Encoders for Text and Graph-Structured Data
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910821296111616 |
|---|---|
| author | Lin, Jiacheng Qian, Kun Han, Haoyu Choudhary, Nurendra Wei, Tianxin Wang, Zhongruo Genc, Sahika Huang, Edward W Wang, Sheng Subbian, Karthik Koutra, Danai Sun, Jimeng |
| author_facet | Lin, Jiacheng Qian, Kun Han, Haoyu Choudhary, Nurendra Wei, Tianxin Wang, Zhongruo Genc, Sahika Huang, Edward W Wang, Sheng Subbian, Karthik Koutra, Danai Sun, Jimeng |
| contents | Graph-structured information offers rich contextual information that can enhance language models by providing structured relationships and hierarchies, leading to more expressive embeddings for various applications such as retrieval, question answering, and classification. However, existing methods for integrating graph and text embeddings, often based on Multi-layer Perceptrons (MLPs) or shallow transformers, are limited in their ability to fully exploit the heterogeneous nature of these modalities. To overcome this, we propose GT2Vec, a simple yet effective framework that leverages Large Language Models (LLMs) to jointly encode text and graph data. Specifically, GT2Vec employs an MLP adapter to project graph embeddings into the same space as text embeddings, allowing the LLM to process both modalities jointly. Unlike prior work, we also introduce contrastive learning to align the graph and text spaces more effectively, thereby improving the quality of learned joint embeddings. Empirical results across six datasets spanning three tasks, knowledge graph-contextualized question answering, graph-text pair classification, and retrieval, demonstrate that GT2Vec consistently outperforms existing baselines, achieving significant improvements across multiple datasets. These results highlight GT2Vec's effectiveness in integrating graph and text data. Ablation studies further validate the effectiveness of our method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_11235 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | GT2Vec: Large Language Models as Multi-Modal Encoders for Text and Graph-Structured Data Lin, Jiacheng Qian, Kun Han, Haoyu Choudhary, Nurendra Wei, Tianxin Wang, Zhongruo Genc, Sahika Huang, Edward W Wang, Sheng Subbian, Karthik Koutra, Danai Sun, Jimeng Computation and Language Graph-structured information offers rich contextual information that can enhance language models by providing structured relationships and hierarchies, leading to more expressive embeddings for various applications such as retrieval, question answering, and classification. However, existing methods for integrating graph and text embeddings, often based on Multi-layer Perceptrons (MLPs) or shallow transformers, are limited in their ability to fully exploit the heterogeneous nature of these modalities. To overcome this, we propose GT2Vec, a simple yet effective framework that leverages Large Language Models (LLMs) to jointly encode text and graph data. Specifically, GT2Vec employs an MLP adapter to project graph embeddings into the same space as text embeddings, allowing the LLM to process both modalities jointly. Unlike prior work, we also introduce contrastive learning to align the graph and text spaces more effectively, thereby improving the quality of learned joint embeddings. Empirical results across six datasets spanning three tasks, knowledge graph-contextualized question answering, graph-text pair classification, and retrieval, demonstrate that GT2Vec consistently outperforms existing baselines, achieving significant improvements across multiple datasets. These results highlight GT2Vec's effectiveness in integrating graph and text data. Ablation studies further validate the effectiveness of our method. |
| title | GT2Vec: Large Language Models as Multi-Modal Encoders for Text and Graph-Structured Data |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.11235 |