Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929592403492864 |
|---|---|
| author | Wang, Yian Qiu, Xiaowen Liu, Jiageng Chen, Zhehuan Cai, Jiting Wang, Yufei Wang, Tsun-Hsuan Xian, Zhou Gan, Chuang |
| author_facet | Wang, Yian Qiu, Xiaowen Liu, Jiageng Chen, Zhehuan Cai, Jiting Wang, Yufei Wang, Tsun-Hsuan Xian, Zhou Gan, Chuang |
| contents | Creating large-scale interactive 3D environments is essential for the development of Robotics and Embodied AI research. Current methods, including manual design, procedural generation, diffusion-based scene generation, and large language model (LLM) guided scene design, are hindered by limitations such as excessive human effort, reliance on predefined rules or training datasets, and limited 3D spatial reasoning ability. Since pre-trained 2D image generative models better capture scene and object configuration than LLMs, we address these challenges by introducing Architect, a generative framework that creates complex and realistic 3D embodied environments leveraging diffusion-based 2D image inpainting. In detail, we utilize foundation visual perception models to obtain each generated object from the image and leverage pre-trained depth estimation models to lift the generated 2D image to 3D space. Our pipeline is further extended to a hierarchical and iterative inpainting process to continuously generate placement of large furniture and small objects to enrich the scene. This iterative structure brings the flexibility for our method to generate or refine scenes from various starting points, such as text, floor plans, or pre-arranged environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_09823 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting Wang, Yian Qiu, Xiaowen Liu, Jiageng Chen, Zhehuan Cai, Jiting Wang, Yufei Wang, Tsun-Hsuan Xian, Zhou Gan, Chuang Computer Vision and Pattern Recognition Creating large-scale interactive 3D environments is essential for the development of Robotics and Embodied AI research. Current methods, including manual design, procedural generation, diffusion-based scene generation, and large language model (LLM) guided scene design, are hindered by limitations such as excessive human effort, reliance on predefined rules or training datasets, and limited 3D spatial reasoning ability. Since pre-trained 2D image generative models better capture scene and object configuration than LLMs, we address these challenges by introducing Architect, a generative framework that creates complex and realistic 3D embodied environments leveraging diffusion-based 2D image inpainting. In detail, we utilize foundation visual perception models to obtain each generated object from the image and leverage pre-trained depth estimation models to lift the generated 2D image to 3D space. Our pipeline is further extended to a hierarchical and iterative inpainting process to continuously generate placement of large furniture and small objects to enrich the scene. This iterative structure brings the flexibility for our method to generate or refine scenes from various starting points, such as text, floor plans, or pre-arranged environments. |
| title | Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2411.09823 |