From Sparse Signal to Smooth Motion: Real-Time Motion Generation with Rolling Prediction Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909569511325696 |
|---|---|
| author | Barquero, German Bertsch, Nadine Marramreddy, Manojkumar Chacón, Carlos Arcadu, Filippo Rigual, Ferran He, Nicky Sijia Palmero, Cristina Escalera, Sergio Ye, Yuting Kips, Robin |
| author_facet | Barquero, German Bertsch, Nadine Marramreddy, Manojkumar Chacón, Carlos Arcadu, Filippo Rigual, Ferran He, Nicky Sijia Palmero, Cristina Escalera, Sergio Ye, Yuting Kips, Robin |
| contents | In extended reality (XR), generating full-body motion of the users is important to understand their actions, drive their virtual avatars for social interaction, and convey a realistic sense of presence. While prior works focused on spatially sparse and always-on input signals from motion controllers, many XR applications opt for vision-based hand tracking for reduced user friction and better immersion. Compared to controllers, hand tracking signals are less accurate and can even be missing for an extended period of time. To handle such unreliable inputs, we present Rolling Prediction Model (RPM), an online and real-time approach that generates smooth full-body motion from temporally and spatially sparse input signals. Our model generates 1) accurate motion that matches the inputs (i.e., tracking mode) and 2) plausible motion when inputs are missing (i.e., synthesis mode). More importantly, RPM generates seamless transitions from tracking to synthesis, and vice versa. To demonstrate the practical importance of handling noisy and missing inputs, we present GORP, the first dataset of realistic sparse inputs from a commercial virtual reality (VR) headset with paired high quality body motion ground truth. GORP provides >14 hours of VR gameplay data from 28 people using motion controllers (spatially sparse) and hand tracking (spatially and temporally sparse). We benchmark RPM against the state of the art on both synthetic data and GORP to highlight how we can bridge the gap for real-world applications with a realistic dataset and by handling unreliable input signals. Our code, pretrained models, and GORP dataset are available in the project webpage. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_05265 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | From Sparse Signal to Smooth Motion: Real-Time Motion Generation with Rolling Prediction Models Barquero, German Bertsch, Nadine Marramreddy, Manojkumar Chacón, Carlos Arcadu, Filippo Rigual, Ferran He, Nicky Sijia Palmero, Cristina Escalera, Sergio Ye, Yuting Kips, Robin Computer Vision and Pattern Recognition In extended reality (XR), generating full-body motion of the users is important to understand their actions, drive their virtual avatars for social interaction, and convey a realistic sense of presence. While prior works focused on spatially sparse and always-on input signals from motion controllers, many XR applications opt for vision-based hand tracking for reduced user friction and better immersion. Compared to controllers, hand tracking signals are less accurate and can even be missing for an extended period of time. To handle such unreliable inputs, we present Rolling Prediction Model (RPM), an online and real-time approach that generates smooth full-body motion from temporally and spatially sparse input signals. Our model generates 1) accurate motion that matches the inputs (i.e., tracking mode) and 2) plausible motion when inputs are missing (i.e., synthesis mode). More importantly, RPM generates seamless transitions from tracking to synthesis, and vice versa. To demonstrate the practical importance of handling noisy and missing inputs, we present GORP, the first dataset of realistic sparse inputs from a commercial virtual reality (VR) headset with paired high quality body motion ground truth. GORP provides >14 hours of VR gameplay data from 28 people using motion controllers (spatially sparse) and hand tracking (spatially and temporally sparse). We benchmark RPM against the state of the art on both synthetic data and GORP to highlight how we can bridge the gap for real-world applications with a realistic dataset and by handling unreliable input signals. Our code, pretrained models, and GORP dataset are available in the project webpage. |
| title | From Sparse Signal to Smooth Motion: Real-Time Motion Generation with Rolling Prediction Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2504.05265 |