Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.14555 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909947002880000 |
|---|---|
| author | Xue, Jintang Zhao, Ganning Yao, Jie-En Chen, Hong-En Hu, Yue Chen, Meida You, Suya Kuo, C. -C. Jay |
| author_facet | Xue, Jintang Zhao, Ganning Yao, Jie-En Chen, Hong-En Hu, Yue Chen, Meida You, Suya Kuo, C. -C. Jay |
| contents | Understanding 3D scenes goes beyond simply recognizing objects; it requires reasoning about the spatial and semantic relationships between them. Current 3D scene-language models often struggle with this relational understanding, particularly when visual embeddings alone do not adequately convey the roles and interactions of objects. In this paper, we introduce Descrip3D, a novel and powerful framework that explicitly encodes the relationships between objects using natural language. Unlike previous methods that rely only on 2D and 3D embeddings, Descrip3D enhances each object with a textual description that captures both its intrinsic attributes and contextual relationships. These relational cues are incorporated into the model through a dual-level integration: embedding fusion and prompt-level injection. This allows for unified reasoning across various tasks such as grounding, captioning, and question answering, all without the need for task-specific heads or additional supervision. When evaluated on five benchmark datasets, including ScanRefer, Multi3DRefer, ScanQA, SQA3D, and Scan2Cap, Descrip3D consistently outperforms strong baseline models, demonstrating the effectiveness of language-guided relational representation for understanding complex indoor scenes. Our code and data are publicly available at https://github.com/jintangxue/Descrip3D. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_14555 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Descrip3D: Enhancing Large Language Model-based 3D Scene Understanding with Object-Level Text Descriptions Xue, Jintang Zhao, Ganning Yao, Jie-En Chen, Hong-En Hu, Yue Chen, Meida You, Suya Kuo, C. -C. Jay Computer Vision and Pattern Recognition Understanding 3D scenes goes beyond simply recognizing objects; it requires reasoning about the spatial and semantic relationships between them. Current 3D scene-language models often struggle with this relational understanding, particularly when visual embeddings alone do not adequately convey the roles and interactions of objects. In this paper, we introduce Descrip3D, a novel and powerful framework that explicitly encodes the relationships between objects using natural language. Unlike previous methods that rely only on 2D and 3D embeddings, Descrip3D enhances each object with a textual description that captures both its intrinsic attributes and contextual relationships. These relational cues are incorporated into the model through a dual-level integration: embedding fusion and prompt-level injection. This allows for unified reasoning across various tasks such as grounding, captioning, and question answering, all without the need for task-specific heads or additional supervision. When evaluated on five benchmark datasets, including ScanRefer, Multi3DRefer, ScanQA, SQA3D, and Scan2Cap, Descrip3D consistently outperforms strong baseline models, demonstrating the effectiveness of language-guided relational representation for understanding complex indoor scenes. Our code and data are publicly available at https://github.com/jintangxue/Descrip3D. |
| title | Descrip3D: Enhancing Large Language Model-based 3D Scene Understanding with Object-Level Text Descriptions |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2507.14555 |