Human-in-the-Loop Local Corrections of 3D Scene Layouts via Infilling
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912511304925184 |
|---|---|
| author | Xie, Christopher Avetisyan, Armen Howard-Jenkins, Henry Siddiqui, Yawar Straub, Julian Newcombe, Richard Balntas, Vasileios Engel, Jakob |
| author_facet | Xie, Christopher Avetisyan, Armen Howard-Jenkins, Henry Siddiqui, Yawar Straub, Julian Newcombe, Richard Balntas, Vasileios Engel, Jakob |
| contents | We present a novel human-in-the-loop approach to estimate 3D scene layout that uses human feedback from an egocentric standpoint. We study this approach through introduction of a novel local correction task, where users identify local errors and prompt a model to automatically correct them. Building on SceneScript, a state-of-the-art framework for 3D scene layout estimation that leverages structured language, we propose a solution that structures this problem as "infilling", a task studied in natural language processing. We train a multi-task version of SceneScript that maintains performance on global predictions while significantly improving its local correction ability. We integrate this into a human-in-the-loop system, enabling a user to iteratively refine scene layout estimates via a low-friction "one-click fix'' workflow. Our system enables the final refined layout to diverge from the training distribution, allowing for more accurate modelling of complex layouts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_11806 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Human-in-the-Loop Local Corrections of 3D Scene Layouts via Infilling Xie, Christopher Avetisyan, Armen Howard-Jenkins, Henry Siddiqui, Yawar Straub, Julian Newcombe, Richard Balntas, Vasileios Engel, Jakob Computer Vision and Pattern Recognition We present a novel human-in-the-loop approach to estimate 3D scene layout that uses human feedback from an egocentric standpoint. We study this approach through introduction of a novel local correction task, where users identify local errors and prompt a model to automatically correct them. Building on SceneScript, a state-of-the-art framework for 3D scene layout estimation that leverages structured language, we propose a solution that structures this problem as "infilling", a task studied in natural language processing. We train a multi-task version of SceneScript that maintains performance on global predictions while significantly improving its local correction ability. We integrate this into a human-in-the-loop system, enabling a user to iteratively refine scene layout estimates via a low-friction "one-click fix'' workflow. Our system enables the final refined layout to diverge from the training distribution, allowing for more accurate modelling of complex layouts. |
| title | Human-in-the-Loop Local Corrections of 3D Scene Layouts via Infilling |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2503.11806 |