InstructLayout: Instruction-Driven 2D and 3D Layout Synthesis with Semantic Graph Prior
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918110697619456 |
|---|---|
| author | Lin, Chenguo Lin, Yuchen Pan, Panwang Zhang, Xuanyang Mu, Yadong |
| author_facet | Lin, Chenguo Lin, Yuchen Pan, Panwang Zhang, Xuanyang Mu, Yadong |
| contents | Comprehending natural language instructions is a charming property for both 2D and 3D layout synthesis systems. Existing methods implicitly model object joint distributions and express object relations, hindering generation's controllability. We introduce InstructLayout, a novel generative framework that integrates a semantic graph prior and a layout decoder to improve controllability and fidelity for 2D and 3D layout synthesis. The proposed semantic graph prior learns layout appearances and object distributions simultaneously, demonstrating versatility across various downstream tasks in a zero-shot manner. To facilitate the benchmarking for text-driven 2D and 3D scene synthesis, we respectively curate two high-quality datasets of layout-instruction pairs from public Internet resources with large language and multimodal models. Extensive experimental results reveal that the proposed method outperforms existing state-of-the-art approaches by a large margin in both 2D and 3D layout synthesis tasks. Thorough ablation studies confirm the efficacy of crucial design components. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_07580 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | InstructLayout: Instruction-Driven 2D and 3D Layout Synthesis with Semantic Graph Prior Lin, Chenguo Lin, Yuchen Pan, Panwang Zhang, Xuanyang Mu, Yadong Computer Vision and Pattern Recognition Comprehending natural language instructions is a charming property for both 2D and 3D layout synthesis systems. Existing methods implicitly model object joint distributions and express object relations, hindering generation's controllability. We introduce InstructLayout, a novel generative framework that integrates a semantic graph prior and a layout decoder to improve controllability and fidelity for 2D and 3D layout synthesis. The proposed semantic graph prior learns layout appearances and object distributions simultaneously, demonstrating versatility across various downstream tasks in a zero-shot manner. To facilitate the benchmarking for text-driven 2D and 3D scene synthesis, we respectively curate two high-quality datasets of layout-instruction pairs from public Internet resources with large language and multimodal models. Extensive experimental results reveal that the proposed method outperforms existing state-of-the-art approaches by a large margin in both 2D and 3D layout synthesis tasks. Thorough ablation studies confirm the efficacy of crucial design components. |
| title | InstructLayout: Instruction-Driven 2D and 3D Layout Synthesis with Semantic Graph Prior |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.07580 |