Open-Universe Indoor Scene Generation using LLM Program Synthesis and Uncurated Object Databases
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917614468464640 |
|---|---|
| author | Aguina-Kang, Rio Gumin, Maxim Han, Do Heon Morris, Stewart Yoo, Seung Jean Ganeshan, Aditya Jones, R. Kenny Wei, Qiuhong Anna Fu, Kailiang Ritchie, Daniel |
| author_facet | Aguina-Kang, Rio Gumin, Maxim Han, Do Heon Morris, Stewart Yoo, Seung Jean Ganeshan, Aditya Jones, R. Kenny Wei, Qiuhong Anna Fu, Kailiang Ritchie, Daniel |
| contents | We present a system for generating indoor scenes in response to text prompts. The prompts are not limited to a fixed vocabulary of scene descriptions, and the objects in generated scenes are not restricted to a fixed set of object categories -- we call this setting indoor scene generation. Unlike most prior work on indoor scene generation, our system does not require a large training dataset of existing 3D scenes. Instead, it leverages the world knowledge encoded in pre-trained large language models (LLMs) to synthesize programs in a domain-specific layout language that describe objects and spatial relations between them. Executing such a program produces a specification of a constraint satisfaction problem, which the system solves using a gradient-based optimization scheme to produce object positions and orientations. To produce object geometry, the system retrieves 3D meshes from a database. Unlike prior work which uses databases of category-annotated, mutually-aligned meshes, we develop a pipeline using vision-language models (VLMs) to retrieve meshes from massive databases of un-annotated, inconsistently-aligned meshes. Experimental evaluations show that our system outperforms generative models trained on 3D data for traditional, closed-universe scene generation tasks; it also outperforms a recent LLM-based layout generation method on open-universe scene generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_09675 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Open-Universe Indoor Scene Generation using LLM Program Synthesis and Uncurated Object Databases Aguina-Kang, Rio Gumin, Maxim Han, Do Heon Morris, Stewart Yoo, Seung Jean Ganeshan, Aditya Jones, R. Kenny Wei, Qiuhong Anna Fu, Kailiang Ritchie, Daniel Computer Vision and Pattern Recognition Graphics We present a system for generating indoor scenes in response to text prompts. The prompts are not limited to a fixed vocabulary of scene descriptions, and the objects in generated scenes are not restricted to a fixed set of object categories -- we call this setting indoor scene generation. Unlike most prior work on indoor scene generation, our system does not require a large training dataset of existing 3D scenes. Instead, it leverages the world knowledge encoded in pre-trained large language models (LLMs) to synthesize programs in a domain-specific layout language that describe objects and spatial relations between them. Executing such a program produces a specification of a constraint satisfaction problem, which the system solves using a gradient-based optimization scheme to produce object positions and orientations. To produce object geometry, the system retrieves 3D meshes from a database. Unlike prior work which uses databases of category-annotated, mutually-aligned meshes, we develop a pipeline using vision-language models (VLMs) to retrieve meshes from massive databases of un-annotated, inconsistently-aligned meshes. Experimental evaluations show that our system outperforms generative models trained on 3D data for traditional, closed-universe scene generation tasks; it also outperforms a recent LLM-based layout generation method on open-universe scene generation. |
| title | Open-Universe Indoor Scene Generation using LLM Program Synthesis and Uncurated Object Databases |
| topic | Computer Vision and Pattern Recognition Graphics |
| url | https://arxiv.org/abs/2403.09675 |