Spatial-ViLT: Enhancing Visual Spatial Reasoning through Multi-Task Learning
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918154023731200 |
|---|---|
| author | Islam, Chashi Mahiul Mamo, Oteo Chacko, Samuel Jacob Liu, Xiuwen Yu, Weikuan |
| author_facet | Islam, Chashi Mahiul Mamo, Oteo Chacko, Samuel Jacob Liu, Xiuwen Yu, Weikuan |
| contents | Vision-language models (VLMs) have advanced multimodal reasoning but still face challenges in spatial reasoning for 3D scenes and complex object configurations. To address this, we introduce SpatialViLT, an enhanced VLM that integrates spatial features like depth maps, 3D coordinates, and edge maps through a multi-task learning framework. This approach enriches multimodal embeddings with spatial understanding. We propose two variants: SpatialViLT and MaskedSpatialViLT, focusing on full and masked object regions, respectively. Additionally, SpatialEnsemble combines both approaches, achieving state-of-the-art accuracy. Our models excel in spatial reasoning categories such as directional, topological, and proximity relations, as demonstrated on the challenging Visual Spatial Reasoning (VSR) dataset. This work represents a significant step in enhancing the spatial intelligence of AI systems, crucial for advanced multimodal understanding and real-world applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_03441 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Spatial-ViLT: Enhancing Visual Spatial Reasoning through Multi-Task Learning Islam, Chashi Mahiul Mamo, Oteo Chacko, Samuel Jacob Liu, Xiuwen Yu, Weikuan Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning 68T45, 68T10, 68T40 Vision-language models (VLMs) have advanced multimodal reasoning but still face challenges in spatial reasoning for 3D scenes and complex object configurations. To address this, we introduce SpatialViLT, an enhanced VLM that integrates spatial features like depth maps, 3D coordinates, and edge maps through a multi-task learning framework. This approach enriches multimodal embeddings with spatial understanding. We propose two variants: SpatialViLT and MaskedSpatialViLT, focusing on full and masked object regions, respectively. Additionally, SpatialEnsemble combines both approaches, achieving state-of-the-art accuracy. Our models excel in spatial reasoning categories such as directional, topological, and proximity relations, as demonstrated on the challenging Visual Spatial Reasoning (VSR) dataset. This work represents a significant step in enhancing the spatial intelligence of AI systems, crucial for advanced multimodal understanding and real-world applications. |
| title | Spatial-ViLT: Enhancing Visual Spatial Reasoning through Multi-Task Learning |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning 68T45, 68T10, 68T40 |
| url | https://arxiv.org/abs/2510.03441 |