PRISM-DP: Spatial Pose-based Observations for Diffusion-Policies via Segmentation, Mesh Generation, and Pose Tracking
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912589962805248 |
|---|---|
| author | Sun, Xiatao Chen, Yinxing Rakita, Daniel |
| author_facet | Sun, Xiatao Chen, Yinxing Rakita, Daniel |
| contents | Diffusion policies generate robot motions by learning to denoise action-space trajectories conditioned on observations. These observations are commonly streams of RGB images, whose high dimensionality includes substantial task-irrelevant information, requiring large models to extract relevant patterns. In contrast, using structured observations like the spatial poses of key objects enables training more compact policies with fewer parameters. However, obtaining accurate object poses in open-set, real-world environments remains challenging, as 6D pose estimation and tracking methods often depend on markers placed on objects beforehand or pre-scanned object meshes that require manual reconstruction. We propose PRISM-DP, an approach that leverages segmentation, mesh generation, and pose tracking models to enable compact diffusion policy learning directly from the spatial poses of task-relevant objects. Crucially, by using a mesh generation model, PRISM-DP eliminates the need for manual mesh creation, improving scalability in open-set environments. Experiments in simulation and the real world show that PRISM-DP outperforms high-dimensional image-based policies and achieves performance comparable to policies trained with ground-truth state information. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_20359 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PRISM-DP: Spatial Pose-based Observations for Diffusion-Policies via Segmentation, Mesh Generation, and Pose Tracking Sun, Xiatao Chen, Yinxing Rakita, Daniel Robotics Diffusion policies generate robot motions by learning to denoise action-space trajectories conditioned on observations. These observations are commonly streams of RGB images, whose high dimensionality includes substantial task-irrelevant information, requiring large models to extract relevant patterns. In contrast, using structured observations like the spatial poses of key objects enables training more compact policies with fewer parameters. However, obtaining accurate object poses in open-set, real-world environments remains challenging, as 6D pose estimation and tracking methods often depend on markers placed on objects beforehand or pre-scanned object meshes that require manual reconstruction. We propose PRISM-DP, an approach that leverages segmentation, mesh generation, and pose tracking models to enable compact diffusion policy learning directly from the spatial poses of task-relevant objects. Crucially, by using a mesh generation model, PRISM-DP eliminates the need for manual mesh creation, improving scalability in open-set environments. Experiments in simulation and the real world show that PRISM-DP outperforms high-dimensional image-based policies and achieves performance comparable to policies trained with ground-truth state information. |
| title | PRISM-DP: Spatial Pose-based Observations for Diffusion-Policies via Segmentation, Mesh Generation, and Pose Tracking |
| topic | Robotics |
| url | https://arxiv.org/abs/2504.20359 |