SceneTeract: Agentic Functional Affordances and VLM Grounding in 3D Scenes
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911558030852096 |
|---|---|
| author | Maillard, Léopold Engelmann, Francis Durand, Tom Pan, Boxiao You, Yang Litany, Or Guibas, Leonidas Ovsjanikov, Maks |
| author_facet | Maillard, Léopold Engelmann, Francis Durand, Tom Pan, Boxiao You, Yang Litany, Or Guibas, Leonidas Ovsjanikov, Maks |
| contents | Embodied AI depends on interactive 3D environments that support meaningful activities for diverse users, yet assessing their functional affordances remains a core challenge. We introduce SceneTeract, a framework that verifies 3D scene functionality under agent-specific constraints. Our core contribution is a grounded verification engine that couples high-level semantic reasoning with low-level geometric checks. SceneTeract decomposes complex activities into sequences of atomic actions and validates each step against accessibility requirements (e.g., reachability, clearance, and navigability) conditioned on an embodied agent profile, using explicit physical and geometric simulations. We deploy SceneTeract to perform an in-depth evaluation of (i) synthetic indoor environments, uncovering frequent functional failures that prevent basic interactions, and (ii) the ability of frontier Vision-Language Models (VLMs) to reason about and predict functional affordances, revealing systematic mismatches between semantic confidence and physical feasibility even for the strongest current models. Finally, we leverage SceneTeract as a reward engine for VLM post-training, enabling scalable distillation of geometric constraints into reasoning models. We release the SceneTeract verification suite and data to bridge perception and physical reality in embodied 3D scene understanding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_29798 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SceneTeract: Agentic Functional Affordances and VLM Grounding in 3D Scenes Maillard, Léopold Engelmann, Francis Durand, Tom Pan, Boxiao You, Yang Litany, Or Guibas, Leonidas Ovsjanikov, Maks Computer Vision and Pattern Recognition Embodied AI depends on interactive 3D environments that support meaningful activities for diverse users, yet assessing their functional affordances remains a core challenge. We introduce SceneTeract, a framework that verifies 3D scene functionality under agent-specific constraints. Our core contribution is a grounded verification engine that couples high-level semantic reasoning with low-level geometric checks. SceneTeract decomposes complex activities into sequences of atomic actions and validates each step against accessibility requirements (e.g., reachability, clearance, and navigability) conditioned on an embodied agent profile, using explicit physical and geometric simulations. We deploy SceneTeract to perform an in-depth evaluation of (i) synthetic indoor environments, uncovering frequent functional failures that prevent basic interactions, and (ii) the ability of frontier Vision-Language Models (VLMs) to reason about and predict functional affordances, revealing systematic mismatches between semantic confidence and physical feasibility even for the strongest current models. Finally, we leverage SceneTeract as a reward engine for VLM post-training, enabling scalable distillation of geometric constraints into reasoning models. We release the SceneTeract verification suite and data to bridge perception and physical reality in embodied 3D scene understanding. |
| title | SceneTeract: Agentic Functional Affordances and VLM Grounding in 3D Scenes |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2603.29798 |