SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916966037454848 |
|---|---|
| author | Qi, Zekun Zhang, Wenyao Ding, Yufei Dong, Runpei Yu, Xinqiang Li, Jingwen Xu, Lingyun Li, Baoyu He, Xialin Fan, Guofan Zhang, Jiazhao He, Jiawei Gu, Jiayuan Jin, Xin Ma, Kaisheng Zhang, Zhizheng Wang, He Yi, Li |
| author_facet | Qi, Zekun Zhang, Wenyao Ding, Yufei Dong, Runpei Yu, Xinqiang Li, Jingwen Xu, Lingyun Li, Baoyu He, Xialin Fan, Guofan Zhang, Jiazhao He, Jiawei Gu, Jiayuan Jin, Xin Ma, Kaisheng Zhang, Zhizheng Wang, He Yi, Li |
| contents | While spatial reasoning has made progress in object localization relationships, it often overlooks object orientation-a key factor in 6-DoF fine-grained manipulation. Traditional pose representations rely on pre-defined frames or templates, limiting generalization and semantic grounding. In this paper, we introduce the concept of semantic orientation, which defines object orientations using natural language in a reference-frame-free manner (e.g., the "plug-in" direction of a USB or the "handle" direction of a cup). To support this, we construct OrienText300K, a large-scale dataset of 3D objects annotated with semantic orientations, and develop PointSO, a general model for zero-shot semantic orientation prediction. By integrating semantic orientation into VLM agents, our SoFar framework enables 6-DoF spatial reasoning and generates robotic actions. Extensive experiments demonstrated the effectiveness and generalization of our SoFar, e.g., zero-shot 48.7% successful rate on Open6DOR and zero-shot 74.9% successful rate on SIMPLER-Env. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_13143 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation Qi, Zekun Zhang, Wenyao Ding, Yufei Dong, Runpei Yu, Xinqiang Li, Jingwen Xu, Lingyun Li, Baoyu He, Xialin Fan, Guofan Zhang, Jiazhao He, Jiawei Gu, Jiayuan Jin, Xin Ma, Kaisheng Zhang, Zhizheng Wang, He Yi, Li Robotics Artificial Intelligence Computer Vision and Pattern Recognition While spatial reasoning has made progress in object localization relationships, it often overlooks object orientation-a key factor in 6-DoF fine-grained manipulation. Traditional pose representations rely on pre-defined frames or templates, limiting generalization and semantic grounding. In this paper, we introduce the concept of semantic orientation, which defines object orientations using natural language in a reference-frame-free manner (e.g., the "plug-in" direction of a USB or the "handle" direction of a cup). To support this, we construct OrienText300K, a large-scale dataset of 3D objects annotated with semantic orientations, and develop PointSO, a general model for zero-shot semantic orientation prediction. By integrating semantic orientation into VLM agents, our SoFar framework enables 6-DoF spatial reasoning and generates robotic actions. Extensive experiments demonstrated the effectiveness and generalization of our SoFar, e.g., zero-shot 48.7% successful rate on Open6DOR and zero-shot 74.9% successful rate on SIMPLER-Env. |
| title | SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation |
| topic | Robotics Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2502.13143 |