SceneMotifCoder: Example-driven Visual Program Learning for Generating 3D Object Arrangements
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910981260574720 |
|---|---|
| author | Tam, Hou In Ivan Pun, Hou In Derek Wang, Austin T. Chang, Angel X. Savva, Manolis |
| author_facet | Tam, Hou In Ivan Pun, Hou In Derek Wang, Austin T. Chang, Angel X. Savva, Manolis |
| contents | Despite advances in text-to-3D generation methods, generation of multi-object arrangements remains challenging. Current methods exhibit failures in generating physically plausible arrangements that respect the provided text description. We present SceneMotifCoder (SMC), an example-driven framework for generating 3D object arrangements through visual program learning. SMC leverages large language models (LLMs) and program synthesis to overcome these challenges by learning visual programs from example arrangements. These programs are generalized into compact, editable meta-programs. When combined with 3D object retrieval and geometry-aware optimization, they can be used to create object arrangements varying in arrangement structure and contained objects. Our experiments show that SMC generates high-quality arrangements using meta-programs learned from few examples. Evaluation results demonstrates that object arrangements generated by SMC better conform to user-specified text descriptions and are more physically plausible when compared with state-of-the-art text-to-3D generation and layout methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_02211 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SceneMotifCoder: Example-driven Visual Program Learning for Generating 3D Object Arrangements Tam, Hou In Ivan Pun, Hou In Derek Wang, Austin T. Chang, Angel X. Savva, Manolis Graphics Despite advances in text-to-3D generation methods, generation of multi-object arrangements remains challenging. Current methods exhibit failures in generating physically plausible arrangements that respect the provided text description. We present SceneMotifCoder (SMC), an example-driven framework for generating 3D object arrangements through visual program learning. SMC leverages large language models (LLMs) and program synthesis to overcome these challenges by learning visual programs from example arrangements. These programs are generalized into compact, editable meta-programs. When combined with 3D object retrieval and geometry-aware optimization, they can be used to create object arrangements varying in arrangement structure and contained objects. Our experiments show that SMC generates high-quality arrangements using meta-programs learned from few examples. Evaluation results demonstrates that object arrangements generated by SMC better conform to user-specified text descriptions and are more physically plausible when compared with state-of-the-art text-to-3D generation and layout methods. |
| title | SceneMotifCoder: Example-driven Visual Program Learning for Generating 3D Object Arrangements |
| topic | Graphics |
| url | https://arxiv.org/abs/2408.02211 |