SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Model
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916167096991744 |
|---|---|
| author | Avetisyan, Armen Xie, Christopher Howard-Jenkins, Henry Yang, Tsun-Yi Aroudj, Samir Patra, Suvam Zhang, Fuyang Frost, Duncan Holland, Luke Orme, Campbell Engel, Jakob Miller, Edward Newcombe, Richard Balntas, Vasileios |
| author_facet | Avetisyan, Armen Xie, Christopher Howard-Jenkins, Henry Yang, Tsun-Yi Aroudj, Samir Patra, Suvam Zhang, Fuyang Frost, Duncan Holland, Luke Orme, Campbell Engel, Jakob Miller, Edward Newcombe, Richard Balntas, Vasileios |
| contents | We introduce SceneScript, a method that directly produces full scene models as a sequence of structured language commands using an autoregressive, token-based approach. Our proposed scene representation is inspired by recent successes in transformers & LLMs, and departs from more traditional methods which commonly describe scenes as meshes, voxel grids, point clouds or radiance fields. Our method infers the set of structured language commands directly from encoded visual data using a scene language encoder-decoder architecture. To train SceneScript, we generate and release a large-scale synthetic dataset called Aria Synthetic Environments consisting of 100k high-quality in-door scenes, with photorealistic and ground-truth annotated renders of egocentric scene walkthroughs. Our method gives state-of-the art results in architectural layout estimation, and competitive results in 3D object detection. Lastly, we explore an advantage for SceneScript, which is the ability to readily adapt to new commands via simple additions to the structured language, which we illustrate for tasks such as coarse 3D object part reconstruction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_13064 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Model Avetisyan, Armen Xie, Christopher Howard-Jenkins, Henry Yang, Tsun-Yi Aroudj, Samir Patra, Suvam Zhang, Fuyang Frost, Duncan Holland, Luke Orme, Campbell Engel, Jakob Miller, Edward Newcombe, Richard Balntas, Vasileios Computer Vision and Pattern Recognition We introduce SceneScript, a method that directly produces full scene models as a sequence of structured language commands using an autoregressive, token-based approach. Our proposed scene representation is inspired by recent successes in transformers & LLMs, and departs from more traditional methods which commonly describe scenes as meshes, voxel grids, point clouds or radiance fields. Our method infers the set of structured language commands directly from encoded visual data using a scene language encoder-decoder architecture. To train SceneScript, we generate and release a large-scale synthetic dataset called Aria Synthetic Environments consisting of 100k high-quality in-door scenes, with photorealistic and ground-truth annotated renders of egocentric scene walkthroughs. Our method gives state-of-the art results in architectural layout estimation, and competitive results in 3D object detection. Lastly, we explore an advantage for SceneScript, which is the ability to readily adapt to new commands via simple additions to the structured language, which we illustrate for tasks such as coarse 3D object part reconstruction. |
| title | SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Model |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.13064 |