EgoMusic-driven Human Dance Motion Estimation with Skeleton Mamba
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913991418183680 |
|---|---|
| author | Nguyen, Quang Le, Nhat Huang, Baoru Vu, Minh Nhat Tang, Chengcheng Nguyen, Van Le, Ngan Vo, Thieu Nguyen, Anh |
| author_facet | Nguyen, Quang Le, Nhat Huang, Baoru Vu, Minh Nhat Tang, Chengcheng Nguyen, Van Le, Ngan Vo, Thieu Nguyen, Anh |
| contents | Estimating human dance motion is a challenging task with various industrial applications. Recently, many efforts have focused on predicting human dance motion using either egocentric video or music as input. However, the task of jointly estimating human motion from both egocentric video and music remains largely unexplored. In this paper, we aim to develop a new method that predicts human dance motion from both egocentric video and music. In practice, the egocentric view often obscures much of the body, making accurate full-pose estimation challenging. Additionally, incorporating music requires the generated head and body movements to align well with both visual and musical inputs. We first introduce EgoAIST++, a new large-scale dataset that combines both egocentric views and music with more than 36 hours of dancing motion. Drawing on the success of diffusion models and Mamba on modeling sequences, we develop an EgoMusic Motion Network with a core Skeleton Mamba that explicitly captures the skeleton structure of the human body. We illustrate that our approach is theoretically supportive. Intensive experiments show that our method clearly outperforms state-of-the-art approaches and generalizes effectively to real-world data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_10522 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EgoMusic-driven Human Dance Motion Estimation with Skeleton Mamba Nguyen, Quang Le, Nhat Huang, Baoru Vu, Minh Nhat Tang, Chengcheng Nguyen, Van Le, Ngan Vo, Thieu Nguyen, Anh Computer Vision and Pattern Recognition Estimating human dance motion is a challenging task with various industrial applications. Recently, many efforts have focused on predicting human dance motion using either egocentric video or music as input. However, the task of jointly estimating human motion from both egocentric video and music remains largely unexplored. In this paper, we aim to develop a new method that predicts human dance motion from both egocentric video and music. In practice, the egocentric view often obscures much of the body, making accurate full-pose estimation challenging. Additionally, incorporating music requires the generated head and body movements to align well with both visual and musical inputs. We first introduce EgoAIST++, a new large-scale dataset that combines both egocentric views and music with more than 36 hours of dancing motion. Drawing on the success of diffusion models and Mamba on modeling sequences, we develop an EgoMusic Motion Network with a core Skeleton Mamba that explicitly captures the skeleton structure of the human body. We illustrate that our approach is theoretically supportive. Intensive experiments show that our method clearly outperforms state-of-the-art approaches and generalizes effectively to real-world data. |
| title | EgoMusic-driven Human Dance Motion Estimation with Skeleton Mamba |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2508.10522 |