RelationField: Relate Anything in Radiance Fields
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912292106403840 |
|---|---|
| author | Koch, Sebastian Wald, Johanna Colosi, Mirco Vaskevicius, Narunas Hermosilla, Pedro Tombari, Federico Ropinski, Timo |
| author_facet | Koch, Sebastian Wald, Johanna Colosi, Mirco Vaskevicius, Narunas Hermosilla, Pedro Tombari, Federico Ropinski, Timo |
| contents | Neural radiance fields are an emerging 3D scene representation and recently even been extended to learn features for scene understanding by distilling open-vocabulary features from vision-language models. However, current method primarily focus on object-centric representations, supporting object segmentation or detection, while understanding semantic relationships between objects remains largely unexplored. To address this gap, we propose RelationField, the first method to extract inter-object relationships directly from neural radiance fields. RelationField represents relationships between objects as pairs of rays within a neural radiance field, effectively extending its formulation to include implicit relationship queries. To teach RelationField complex, open-vocabulary relationships, relationship knowledge is distilled from multi-modal LLMs. To evaluate RelationField, we solve open-vocabulary 3D scene graph generation tasks and relationship-guided instance segmentation, achieving state-of-the-art performance in both tasks. See the project website at https://relationfield.github.io. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_13652 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | RelationField: Relate Anything in Radiance Fields Koch, Sebastian Wald, Johanna Colosi, Mirco Vaskevicius, Narunas Hermosilla, Pedro Tombari, Federico Ropinski, Timo Computer Vision and Pattern Recognition Neural radiance fields are an emerging 3D scene representation and recently even been extended to learn features for scene understanding by distilling open-vocabulary features from vision-language models. However, current method primarily focus on object-centric representations, supporting object segmentation or detection, while understanding semantic relationships between objects remains largely unexplored. To address this gap, we propose RelationField, the first method to extract inter-object relationships directly from neural radiance fields. RelationField represents relationships between objects as pairs of rays within a neural radiance field, effectively extending its formulation to include implicit relationship queries. To teach RelationField complex, open-vocabulary relationships, relationship knowledge is distilled from multi-modal LLMs. To evaluate RelationField, we solve open-vocabulary 3D scene graph generation tasks and relationship-guided instance segmentation, achieving state-of-the-art performance in both tasks. See the project website at https://relationfield.github.io. |
| title | RelationField: Relate Anything in Radiance Fields |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.13652 |