Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.03642 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912593863507968 |
|---|---|
| author | Zhang, Haoyu Liu, Meng Li, Zaijing Wen, Haokun Guan, Weili Wang, Yaowei Nie, Liqiang |
| author_facet | Zhang, Haoyu Liu, Meng Li, Zaijing Wen, Haokun Guan, Weili Wang, Yaowei Nie, Liqiang |
| contents | Visual-spatial understanding, the ability to infer object relationships and layouts from visual input, is fundamental to downstream tasks such as robotic navigation and embodied interaction. However, existing methods face spatial uncertainty and data scarcity, limiting the 3D spatial reasoning capability of pre-trained vision-language models (VLMs). To address these challenges, we present a unified framework for enhancing 3D spatial reasoning in pre-trained VLMs without modifying their architecture. This framework combines SpatialMind, a structured prompting strategy that decomposes complex scenes and questions into interpretable reasoning steps, with ScanForgeQA, a scalable question-answering dataset built from diverse 3D simulation scenes through an automated construction process designed for fine-tuning. Extensive experiments across multiple benchmarks demonstrate the individual and combined effectiveness of our prompting and fine-tuning strategies, and yield insights that may inspire future research on visual-spatial understanding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_03642 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Spatial Understanding from Videos: Structured Prompts Meet Simulation Data Zhang, Haoyu Liu, Meng Li, Zaijing Wen, Haokun Guan, Weili Wang, Yaowei Nie, Liqiang Computer Vision and Pattern Recognition Artificial Intelligence Visual-spatial understanding, the ability to infer object relationships and layouts from visual input, is fundamental to downstream tasks such as robotic navigation and embodied interaction. However, existing methods face spatial uncertainty and data scarcity, limiting the 3D spatial reasoning capability of pre-trained vision-language models (VLMs). To address these challenges, we present a unified framework for enhancing 3D spatial reasoning in pre-trained VLMs without modifying their architecture. This framework combines SpatialMind, a structured prompting strategy that decomposes complex scenes and questions into interpretable reasoning steps, with ScanForgeQA, a scalable question-answering dataset built from diverse 3D simulation scenes through an automated construction process designed for fine-tuning. Extensive experiments across multiple benchmarks demonstrate the individual and combined effectiveness of our prompting and fine-tuning strategies, and yield insights that may inspire future research on visual-spatial understanding. |
| title | Spatial Understanding from Videos: Structured Prompts Meet Simulation Data |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2506.03642 |