MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909649424351232 |
|---|---|
| author | Xu, Zunnan Lin, Yukang Han, Haonan Yang, Sicheng Li, Ronghui Zhang, Yachao Li, Xiu |
| author_facet | Xu, Zunnan Lin, Yukang Han, Haonan Yang, Sicheng Li, Ronghui Zhang, Yachao Li, Xiu |
| contents | Gesture synthesis is a vital realm of human-computer interaction, with wide-ranging applications across various fields like film, robotics, and virtual reality. Recent advancements have utilized the diffusion model and attention mechanisms to improve gesture synthesis. However, due to the high computational complexity of these techniques, generating long and diverse sequences with low latency remains a challenge. We explore the potential of state space models (SSMs) to address the challenge, implementing a two-stage modeling strategy with discrete motion priors to enhance the quality of gestures. Leveraging the foundational Mamba block, we introduce MambaTalk, enhancing gesture diversity and rhythm through multimodal integration. Extensive experiments demonstrate that our method matches or exceeds the performance of state-of-the-art models. Our project is publicly available at https://kkakkkka.github.io/MambaTalk |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_09471 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Models Xu, Zunnan Lin, Yukang Han, Haonan Yang, Sicheng Li, Ronghui Zhang, Yachao Li, Xiu Computer Vision and Pattern Recognition Human-Computer Interaction Gesture synthesis is a vital realm of human-computer interaction, with wide-ranging applications across various fields like film, robotics, and virtual reality. Recent advancements have utilized the diffusion model and attention mechanisms to improve gesture synthesis. However, due to the high computational complexity of these techniques, generating long and diverse sequences with low latency remains a challenge. We explore the potential of state space models (SSMs) to address the challenge, implementing a two-stage modeling strategy with discrete motion priors to enhance the quality of gestures. Leveraging the foundational Mamba block, we introduce MambaTalk, enhancing gesture diversity and rhythm through multimodal integration. Extensive experiments demonstrate that our method matches or exceeds the performance of state-of-the-art models. Our project is publicly available at https://kkakkkka.github.io/MambaTalk |
| title | MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Models |
| topic | Computer Vision and Pattern Recognition Human-Computer Interaction |
| url | https://arxiv.org/abs/2403.09471 |