EgoDemoGen: Egocentric Demonstration Generation for Viewpoint Generalization in Robotic Manipulation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914431865192448 |
|---|---|
| author | Xu, Yuan Yang, Jiabing Wang, Xiaofeng Chen, Yixiang Zhu, Zheng Fang, Bowen Huang, Guan Chen, Xinze Ye, Yun Zhang, Qiang Li, Peiyan Wu, Xiangnan Wang, Kai Zhan, Bing Lu, Shuo Liu, Jing Liu, Nianfeng Huang, Yan Wang, Liang |
| author_facet | Xu, Yuan Yang, Jiabing Wang, Xiaofeng Chen, Yixiang Zhu, Zheng Fang, Bowen Huang, Guan Chen, Xinze Ye, Yun Zhang, Qiang Li, Peiyan Wu, Xiangnan Wang, Kai Zhan, Bing Lu, Shuo Liu, Jing Liu, Nianfeng Huang, Yan Wang, Liang |
| contents | Imitation learning based visuomotor policies have achieved strong performance in robotic manipulation, yet they often remain sensitive to egocentric viewpoint shifts. Unlike third-person viewpoint changes that only move the camera, egocentric shifts simultaneously alter both the camera pose and the robot action coordinate frame, making it necessary to jointly transfer action trajectories and synthesize corresponding observations under novel egocentric viewpoints. To address this challenge, we present EgoDemoGen, a framework that generates paired observation--action demonstrations under novel egocentric viewpoints through two key components: 1{)} EgoTrajTransfer, which transfers robot trajectories to the novel egocentric coordinate frame through motion-skill segmentation, geometry-aware transformation, and inverse kinematics filtering; and 2{)} EgoViewTransfer, a conditional video generation model that fuses a novel-viewpoint reprojected scene video and a robot motion video rendered from the transferred trajectory to synthesize photorealistic observations, trained with a self-supervised double reprojection strategy without requiring multi-viewpoint data. Experiments in simulation and real-world settings show that EgoDemoGen consistently improves policy success rates under both standard and novel egocentric viewpoints, with absolute gains of +24.6\% and +16.9\% in simulation and +16.0\% and +23.0\% on the real robot. Moreover, EgoViewTransfer achieves superior video generation quality for novel egocentric observations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_22578 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EgoDemoGen: Egocentric Demonstration Generation for Viewpoint Generalization in Robotic Manipulation Xu, Yuan Yang, Jiabing Wang, Xiaofeng Chen, Yixiang Zhu, Zheng Fang, Bowen Huang, Guan Chen, Xinze Ye, Yun Zhang, Qiang Li, Peiyan Wu, Xiangnan Wang, Kai Zhan, Bing Lu, Shuo Liu, Jing Liu, Nianfeng Huang, Yan Wang, Liang Robotics Imitation learning based visuomotor policies have achieved strong performance in robotic manipulation, yet they often remain sensitive to egocentric viewpoint shifts. Unlike third-person viewpoint changes that only move the camera, egocentric shifts simultaneously alter both the camera pose and the robot action coordinate frame, making it necessary to jointly transfer action trajectories and synthesize corresponding observations under novel egocentric viewpoints. To address this challenge, we present EgoDemoGen, a framework that generates paired observation--action demonstrations under novel egocentric viewpoints through two key components: 1{)} EgoTrajTransfer, which transfers robot trajectories to the novel egocentric coordinate frame through motion-skill segmentation, geometry-aware transformation, and inverse kinematics filtering; and 2{)} EgoViewTransfer, a conditional video generation model that fuses a novel-viewpoint reprojected scene video and a robot motion video rendered from the transferred trajectory to synthesize photorealistic observations, trained with a self-supervised double reprojection strategy without requiring multi-viewpoint data. Experiments in simulation and real-world settings show that EgoDemoGen consistently improves policy success rates under both standard and novel egocentric viewpoints, with absolute gains of +24.6\% and +16.9\% in simulation and +16.0\% and +23.0\% on the real robot. Moreover, EgoViewTransfer achieves superior video generation quality for novel egocentric observations. |
| title | EgoDemoGen: Egocentric Demonstration Generation for Viewpoint Generalization in Robotic Manipulation |
| topic | Robotics |
| url | https://arxiv.org/abs/2509.22578 |