HMD^2: Environment-aware Motion Generation from Single Egocentric Head-Mounted Device
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915177980493824 |
|---|---|
| author | Guzov, Vladimir Jiang, Yifeng Hong, Fangzhou Pons-Moll, Gerard Newcombe, Richard Liu, C. Karen Ye, Yuting Ma, Lingni |
| author_facet | Guzov, Vladimir Jiang, Yifeng Hong, Fangzhou Pons-Moll, Gerard Newcombe, Richard Liu, C. Karen Ye, Yuting Ma, Lingni |
| contents | This paper investigates the generation of realistic full-body human motion using a single head-mounted device with an outward-facing color camera and the ability to perform visual SLAM. To address the ambiguity of this setup, we present HMD^2, a novel system that balances motion reconstruction and generation. From a reconstruction standpoint, it aims to maximally utilize the camera streams to produce both analytical and learned features, including head motion, SLAM point cloud, and image embeddings. On the generative front, HMD^2 employs a multi-modal conditional motion diffusion model with a Transformer backbone to maintain temporal coherence of generated motions, and utilizes autoregressive inpainting to facilitate online motion inference with minimal latency (0.17 seconds). We show that our system provides an effective and robust solution that scales to a diverse dataset of over 200 hours of motion in complex indoor and outdoor environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_13426 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | HMD^2: Environment-aware Motion Generation from Single Egocentric Head-Mounted Device Guzov, Vladimir Jiang, Yifeng Hong, Fangzhou Pons-Moll, Gerard Newcombe, Richard Liu, C. Karen Ye, Yuting Ma, Lingni Computer Vision and Pattern Recognition This paper investigates the generation of realistic full-body human motion using a single head-mounted device with an outward-facing color camera and the ability to perform visual SLAM. To address the ambiguity of this setup, we present HMD^2, a novel system that balances motion reconstruction and generation. From a reconstruction standpoint, it aims to maximally utilize the camera streams to produce both analytical and learned features, including head motion, SLAM point cloud, and image embeddings. On the generative front, HMD^2 employs a multi-modal conditional motion diffusion model with a Transformer backbone to maintain temporal coherence of generated motions, and utilizes autoregressive inpainting to facilitate online motion inference with minimal latency (0.17 seconds). We show that our system provides an effective and robust solution that scales to a diverse dataset of over 200 hours of motion in complex indoor and outdoor environments. |
| title | HMD^2: Environment-aware Motion Generation from Single Egocentric Head-Mounted Device |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2409.13426 |