MaskControl: Spatio-Temporal Control for Masked Motion Synthesis
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914100228915200 |
|---|---|
| author | Pinyoanuntapong, Ekkasit Saleem, Muhammad Usama Karunratanakul, Korrawe Wang, Pu Xue, Hongfei Chen, Chen Guo, Chuan Cao, Junli Ren, Jian Tulyakov, Sergey |
| author_facet | Pinyoanuntapong, Ekkasit Saleem, Muhammad Usama Karunratanakul, Korrawe Wang, Pu Xue, Hongfei Chen, Chen Guo, Chuan Cao, Junli Ren, Jian Tulyakov, Sergey |
| contents | Recent advances in motion diffusion models have enabled spatially controllable text-to-motion generation. However, these models struggle to achieve high-precision control while maintaining high-quality motion generation. To address these challenges, we propose MaskControl, the first approach to introduce controllability to the generative masked motion model. Our approach introduces two key innovations. First, \textit{Logits Regularizer} implicitly perturbs logits at training time to align the distribution of motion tokens with the controlled joint positions, while regularizing the categorical token prediction to ensure high-fidelity generation. Second, \textit{Logit Optimization} explicitly optimizes the predicted logits during inference time, directly reshaping the token distribution that forces the generated motion to accurately align with the controlled joint positions. Moreover, we introduce \textit{Differentiable Expectation Sampling (DES)} to combat the non-differential distribution sampling process encountered by logits regularizer and optimization. Extensive experiments demonstrate that MaskControl outperforms state-of-the-art methods, achieving superior motion quality (FID decreases by ~77\%) and higher control precision (average error 0.91 vs. 1.08). Additionally, MaskControl enables diverse applications, including any-joint-any-frame control, body-part timeline control, and zero-shot objective control. Video visualization can be found at https://www.ekkasit.com/ControlMM-page/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_10780 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MaskControl: Spatio-Temporal Control for Masked Motion Synthesis Pinyoanuntapong, Ekkasit Saleem, Muhammad Usama Karunratanakul, Korrawe Wang, Pu Xue, Hongfei Chen, Chen Guo, Chuan Cao, Junli Ren, Jian Tulyakov, Sergey Computer Vision and Pattern Recognition Recent advances in motion diffusion models have enabled spatially controllable text-to-motion generation. However, these models struggle to achieve high-precision control while maintaining high-quality motion generation. To address these challenges, we propose MaskControl, the first approach to introduce controllability to the generative masked motion model. Our approach introduces two key innovations. First, \textit{Logits Regularizer} implicitly perturbs logits at training time to align the distribution of motion tokens with the controlled joint positions, while regularizing the categorical token prediction to ensure high-fidelity generation. Second, \textit{Logit Optimization} explicitly optimizes the predicted logits during inference time, directly reshaping the token distribution that forces the generated motion to accurately align with the controlled joint positions. Moreover, we introduce \textit{Differentiable Expectation Sampling (DES)} to combat the non-differential distribution sampling process encountered by logits regularizer and optimization. Extensive experiments demonstrate that MaskControl outperforms state-of-the-art methods, achieving superior motion quality (FID decreases by ~77\%) and higher control precision (average error 0.91 vs. 1.08). Additionally, MaskControl enables diverse applications, including any-joint-any-frame control, body-part timeline control, and zero-shot objective control. Video visualization can be found at https://www.ekkasit.com/ControlMM-page/ |
| title | MaskControl: Spatio-Temporal Control for Masked Motion Synthesis |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.10780 |