InstructG2I: Synthesizing Images from Multimodal Attributed Graphs
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912065329823744 |
|---|---|
| author | Jin, Bowen Pang, Ziqi Guo, Bingjun Wang, Yu-Xiong You, Jiaxuan Han, Jiawei |
| author_facet | Jin, Bowen Pang, Ziqi Guo, Bingjun Wang, Yu-Xiong You, Jiaxuan Han, Jiawei |
| contents | In this paper, we approach an overlooked yet critical task Graph2Image: generating images from multimodal attributed graphs (MMAGs). This task poses significant challenges due to the explosion in graph size, dependencies among graph entities, and the need for controllability in graph conditions. To address these challenges, we propose a graph context-conditioned diffusion model called InstructG2I. InstructG2I first exploits the graph structure and multimodal information to conduct informative neighbor sampling by combining personalized page rank and re-ranking based on vision-language features. Then, a Graph-QFormer encoder adaptively encodes the graph nodes into an auxiliary set of graph prompts to guide the denoising process of diffusion. Finally, we propose graph classifier-free guidance, enabling controllable generation by varying the strength of graph guidance and multiple connected edges to a node. Extensive experiments conducted on three datasets from different domains demonstrate the effectiveness and controllability of our approach. The code is available at https://github.com/PeterGriffinJin/InstructG2I. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_07157 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | InstructG2I: Synthesizing Images from Multimodal Attributed Graphs Jin, Bowen Pang, Ziqi Guo, Bingjun Wang, Yu-Xiong You, Jiaxuan Han, Jiawei Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Machine Learning Social and Information Networks In this paper, we approach an overlooked yet critical task Graph2Image: generating images from multimodal attributed graphs (MMAGs). This task poses significant challenges due to the explosion in graph size, dependencies among graph entities, and the need for controllability in graph conditions. To address these challenges, we propose a graph context-conditioned diffusion model called InstructG2I. InstructG2I first exploits the graph structure and multimodal information to conduct informative neighbor sampling by combining personalized page rank and re-ranking based on vision-language features. Then, a Graph-QFormer encoder adaptively encodes the graph nodes into an auxiliary set of graph prompts to guide the denoising process of diffusion. Finally, we propose graph classifier-free guidance, enabling controllable generation by varying the strength of graph guidance and multiple connected edges to a node. Extensive experiments conducted on three datasets from different domains demonstrate the effectiveness and controllability of our approach. The code is available at https://github.com/PeterGriffinJin/InstructG2I. |
| title | InstructG2I: Synthesizing Images from Multimodal Attributed Graphs |
| topic | Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Machine Learning Social and Information Networks |
| url | https://arxiv.org/abs/2410.07157 |