Proxy3D: Efficient 3D Representations for Vision-Language Models via Semantic Clustering and Alignment
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910202131906560 |
|---|---|
| author | Jiang, Jerry Sun, Haowen Gudovskiy, Denis Nakata, Yohei Okuno, Tomoyuki Keutzer, Kurt Zheng, Wenzhao |
| author_facet | Jiang, Jerry Sun, Haowen Gudovskiy, Denis Nakata, Yohei Okuno, Tomoyuki Keutzer, Kurt Zheng, Wenzhao |
| contents | Spatial intelligence in vision-language models (VLMs) attracts research interest with the practical demand to reason in the 3D world.Despite promising results, most existing methods follow the conventional 2D pipeline in VLMs and use pixel-aligned representations for the vision modality. However, correspondence-based models with implicit 3D scene understanding often fail to achieve spatial consistency, and representation-based models with 3D geometric priors lack efficiency in vision sequence serialization. To address this, we propose a Proxy3D method with compact yet comprehensive 3D proxy representations for the vision modality. Given only video frames as input, we employ semantic and geometric encoders to extract scene features and then perform their semantic-aware clustering to obtain a set of proxies in the 3D space. For representation alignment, we further curate the SpaceSpan dataset and apply multi-stage training to adopt the proposed 3D proxy representations with the VLM. When using shorter sequences for vision information, our method achieves competitive or state-of-the-art performance in 3D visual question answering, visual grounding and general spatial intelligence benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_08064 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Proxy3D: Efficient 3D Representations for Vision-Language Models via Semantic Clustering and Alignment Jiang, Jerry Sun, Haowen Gudovskiy, Denis Nakata, Yohei Okuno, Tomoyuki Keutzer, Kurt Zheng, Wenzhao Computer Vision and Pattern Recognition Spatial intelligence in vision-language models (VLMs) attracts research interest with the practical demand to reason in the 3D world.Despite promising results, most existing methods follow the conventional 2D pipeline in VLMs and use pixel-aligned representations for the vision modality. However, correspondence-based models with implicit 3D scene understanding often fail to achieve spatial consistency, and representation-based models with 3D geometric priors lack efficiency in vision sequence serialization. To address this, we propose a Proxy3D method with compact yet comprehensive 3D proxy representations for the vision modality. Given only video frames as input, we employ semantic and geometric encoders to extract scene features and then perform their semantic-aware clustering to obtain a set of proxies in the 3D space. For representation alignment, we further curate the SpaceSpan dataset and apply multi-stage training to adopt the proposed 3D proxy representations with the VLM. When using shorter sequences for vision information, our method achieves competitive or state-of-the-art performance in 3D visual question answering, visual grounding and general spatial intelligence benchmarks. |
| title | Proxy3D: Efficient 3D Representations for Vision-Language Models via Semantic Clustering and Alignment |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2605.08064 |