EgoPoseFormer v2: Accurate Egocentric Human Motion Estimation for AR/VR
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915834043039744 |
|---|---|
| author | Li, Zhenyu Dwivedi, Sai Kumar Maric, Filip Chacon, Carlos Bertsch, Nadine Arcadu, Filippo Hodan, Tomas Ramamonjisoa, Michael Wonka, Peter Zhao, Amy Kips, Robin Keskin, Cem Tkach, Anastasia Yang, Chenhongyi |
| author_facet | Li, Zhenyu Dwivedi, Sai Kumar Maric, Filip Chacon, Carlos Bertsch, Nadine Arcadu, Filippo Hodan, Tomas Ramamonjisoa, Michael Wonka, Peter Zhao, Amy Kips, Robin Keskin, Cem Tkach, Anastasia Yang, Chenhongyi |
| contents | Egocentric human motion estimation is essential for AR/VR experiences, yet remains challenging due to limited body coverage from the egocentric viewpoint, frequent occlusions, and scarce labeled data. We present EgoPoseFormer v2, a method that addresses these challenges through two key contributions: (1) a transformer-based model for temporally consistent and spatially grounded body pose estimation, and (2) an auto-labeling system that enables the use of large unlabeled datasets for training. Our model is fully differentiable, introduces identity-conditioned queries, multi-view spatial refinement, causal temporal attention, and supports both keypoints and parametric body representations under a constant compute budget. The auto-labeling system scales learning to tens of millions of unlabeled frames via uncertainty-aware semi-supervised training. The system follows a teacher-student schema to generate pseudo-labels and guide training with uncertainty distillation, enabling the model to generalize to different environments. On the EgoBody3M benchmark, with a 0.8 ms latency on GPU, our model outperforms two state-of-the-art methods by 12.2% and 19.4% in accuracy, and reduces temporal jitter by 22.2% and 51.7%. Furthermore, our auto-labeling system further improves the wrist MPJPE by 13.1%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_04090 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | EgoPoseFormer v2: Accurate Egocentric Human Motion Estimation for AR/VR Li, Zhenyu Dwivedi, Sai Kumar Maric, Filip Chacon, Carlos Bertsch, Nadine Arcadu, Filippo Hodan, Tomas Ramamonjisoa, Michael Wonka, Peter Zhao, Amy Kips, Robin Keskin, Cem Tkach, Anastasia Yang, Chenhongyi Computer Vision and Pattern Recognition Graphics Human-Computer Interaction Egocentric human motion estimation is essential for AR/VR experiences, yet remains challenging due to limited body coverage from the egocentric viewpoint, frequent occlusions, and scarce labeled data. We present EgoPoseFormer v2, a method that addresses these challenges through two key contributions: (1) a transformer-based model for temporally consistent and spatially grounded body pose estimation, and (2) an auto-labeling system that enables the use of large unlabeled datasets for training. Our model is fully differentiable, introduces identity-conditioned queries, multi-view spatial refinement, causal temporal attention, and supports both keypoints and parametric body representations under a constant compute budget. The auto-labeling system scales learning to tens of millions of unlabeled frames via uncertainty-aware semi-supervised training. The system follows a teacher-student schema to generate pseudo-labels and guide training with uncertainty distillation, enabling the model to generalize to different environments. On the EgoBody3M benchmark, with a 0.8 ms latency on GPU, our model outperforms two state-of-the-art methods by 12.2% and 19.4% in accuracy, and reduces temporal jitter by 22.2% and 51.7%. Furthermore, our auto-labeling system further improves the wrist MPJPE by 13.1%. |
| title | EgoPoseFormer v2: Accurate Egocentric Human Motion Estimation for AR/VR |
| topic | Computer Vision and Pattern Recognition Graphics Human-Computer Interaction |
| url | https://arxiv.org/abs/2603.04090 |