Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2506.23329 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909665689862144 |
|---|---|
| author | Liu, Parker Li, Chenxin Li, Zhengxin Wu, Yipeng Li, Wuyang Yang, Zhiqin Zhang, Zhenyuan Lin, Yunlong Han, Sirui Feng, Brandon Y. |
| author_facet | Liu, Parker Li, Chenxin Li, Zhengxin Wu, Yipeng Li, Wuyang Yang, Zhiqin Zhang, Zhenyuan Lin, Yunlong Han, Sirui Feng, Brandon Y. |
| contents | Vision-language models (VLMs) excel at descriptive tasks, but whether they truly understand scenes from visual observations remains uncertain. We introduce IR3D-Bench, a benchmark challenging VLMs to demonstrate understanding through active creation rather than passive recognition. Grounded in the analysis-by-synthesis paradigm, IR3D-Bench tasks Vision-Language Agents (VLAs) with actively using programming and rendering tools to recreate the underlying 3D structure of an input image, achieving agentic inverse rendering through tool use. This "understanding-by-creating" approach probes the tool-using generative capacity of VLAs, moving beyond the descriptive or conversational capacity measured by traditional scene understanding benchmarks. We provide a comprehensive suite of metrics to evaluate geometric accuracy, spatial relations, appearance attributes, and overall plausibility. Initial experiments on agentic inverse rendering powered by various state-of-the-art VLMs highlight current limitations, particularly in visual precision rather than basic tool usage. IR3D-Bench, including data and evaluation protocols, is released to facilitate systematic study and development of tool-using VLAs towards genuine scene understanding by creating. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_23329 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering Liu, Parker Li, Chenxin Li, Zhengxin Wu, Yipeng Li, Wuyang Yang, Zhiqin Zhang, Zhenyuan Lin, Yunlong Han, Sirui Feng, Brandon Y. Computer Vision and Pattern Recognition Vision-language models (VLMs) excel at descriptive tasks, but whether they truly understand scenes from visual observations remains uncertain. We introduce IR3D-Bench, a benchmark challenging VLMs to demonstrate understanding through active creation rather than passive recognition. Grounded in the analysis-by-synthesis paradigm, IR3D-Bench tasks Vision-Language Agents (VLAs) with actively using programming and rendering tools to recreate the underlying 3D structure of an input image, achieving agentic inverse rendering through tool use. This "understanding-by-creating" approach probes the tool-using generative capacity of VLAs, moving beyond the descriptive or conversational capacity measured by traditional scene understanding benchmarks. We provide a comprehensive suite of metrics to evaluate geometric accuracy, spatial relations, appearance attributes, and overall plausibility. Initial experiments on agentic inverse rendering powered by various state-of-the-art VLMs highlight current limitations, particularly in visual precision rather than basic tool usage. IR3D-Bench, including data and evaluation protocols, is released to facilitate systematic study and development of tool-using VLAs towards genuine scene understanding by creating. |
| title | IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2506.23329 |