ImitDiff: Transferring Foundation-Model Priors for Distraction Robust Visuomotor Policy
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918191306899456 |
|---|---|
| author | Dong, Yuhang Ge, Haizhou Zeng, Yupei Zhang, Jiangning Tian, Beiwen Zhu, Hongrui Jia, Yufei Wang, Ruixiang Xue, Zhucun Zhou, Guyue Ma, Longhua Tian, Guanzhong |
| author_facet | Dong, Yuhang Ge, Haizhou Zeng, Yupei Zhang, Jiangning Tian, Beiwen Zhu, Hongrui Jia, Yufei Wang, Ruixiang Xue, Zhucun Zhou, Guyue Ma, Longhua Tian, Guanzhong |
| contents | Visuomotor imitation learning policies enable robots to efficiently acquire manipulation skills from visual demonstrations. However, as scene complexity and visual distractions increase, policies that perform well in simple settings often experience substantial performance degradation. To address this challenge, we propose ImitDiff, a diffusion-based imitation learning policy guided by fine-grained semantics within a dual-resolution workflow. Leveraging pretrained priors of vision-language foundation models, our method transforms high-level instructions into pixel-level visual semantic masks. These masks guide a dual-resolution perception pipeline that captures both global context (e.g., overall layout) from low-resolution observation and fine-grained local features (e.g., geometric details) from high-resolution observation, enabling the policy to focus on task-relevant regions. Additionally, we introduce a consistency-driven diffusion transformer action head that bridges visual semantic conditions and real-time action generation. Extensive experiments demonstrate that ImitDiff outperforms state-of-the-art vision-language manipulation frameworks, as well as visuomotor imitation learning policies, particularly under increased scene complexity and visual distractions. Notably, ImitDiff exhibits strong generalization in zero-shot settings involving novel objects and visual distractions. Furthermore, our consistency-driven action head achieves an order-of-magnitude improvement in inference speed while maintaining competitive success rates. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_09649 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ImitDiff: Transferring Foundation-Model Priors for Distraction Robust Visuomotor Policy Dong, Yuhang Ge, Haizhou Zeng, Yupei Zhang, Jiangning Tian, Beiwen Zhu, Hongrui Jia, Yufei Wang, Ruixiang Xue, Zhucun Zhou, Guyue Ma, Longhua Tian, Guanzhong Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Robotics Visuomotor imitation learning policies enable robots to efficiently acquire manipulation skills from visual demonstrations. However, as scene complexity and visual distractions increase, policies that perform well in simple settings often experience substantial performance degradation. To address this challenge, we propose ImitDiff, a diffusion-based imitation learning policy guided by fine-grained semantics within a dual-resolution workflow. Leveraging pretrained priors of vision-language foundation models, our method transforms high-level instructions into pixel-level visual semantic masks. These masks guide a dual-resolution perception pipeline that captures both global context (e.g., overall layout) from low-resolution observation and fine-grained local features (e.g., geometric details) from high-resolution observation, enabling the policy to focus on task-relevant regions. Additionally, we introduce a consistency-driven diffusion transformer action head that bridges visual semantic conditions and real-time action generation. Extensive experiments demonstrate that ImitDiff outperforms state-of-the-art vision-language manipulation frameworks, as well as visuomotor imitation learning policies, particularly under increased scene complexity and visual distractions. Notably, ImitDiff exhibits strong generalization in zero-shot settings involving novel objects and visual distractions. Furthermore, our consistency-driven action head achieves an order-of-magnitude improvement in inference speed while maintaining competitive success rates. |
| title | ImitDiff: Transferring Foundation-Model Priors for Distraction Robust Visuomotor Policy |
| topic | Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Robotics |
| url | https://arxiv.org/abs/2502.09649 |