LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913724839755776 |
|---|---|
| author | Li, Zhe Yuan, Weihao He, Yisheng Qiu, Lingteng Zhu, Shenhao Gu, Xiaodong Shen, Weichao Dong, Yuan Dong, Zilong Yang, Laurence T. |
| author_facet | Li, Zhe Yuan, Weihao He, Yisheng Qiu, Lingteng Zhu, Shenhao Gu, Xiaodong Shen, Weichao Dong, Yuan Dong, Zilong Yang, Laurence T. |
| contents | Language plays a vital role in the realm of human motion. Existing methods have largely depended on CLIP text embeddings for motion generation, yet they fall short in effectively aligning language and motion due to CLIP's pretraining on static image-text pairs. This work introduces LaMP, a novel Language-Motion Pretraining model, which transitions from a language-vision to a more suitable language-motion latent space. It addresses key limitations by generating motion-informative text embeddings, significantly enhancing the relevance and semantics of generated motion sequences. With LaMP, we advance three key tasks: text-to-motion generation, motion-text retrieval, and motion captioning through aligned language-motion representation learning. For generation, we utilize LaMP to provide the text condition instead of CLIP, and an autoregressive masked prediction is designed to achieve mask modeling without rank collapse in transformers. For retrieval, motion features from LaMP's motion transformer interact with query tokens to retrieve text features from the text transformer, and vice versa. For captioning, we finetune a large language model with the language-informative motion features to develop a strong motion captioning model. In addition, we introduce the LaMP-BertScore metric to assess the alignment of generated motions with textual descriptions. Extensive experimental results on multiple datasets demonstrate substantial improvements over previous methods across all three tasks. The code of our method will be made public. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_07093 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioning Li, Zhe Yuan, Weihao He, Yisheng Qiu, Lingteng Zhu, Shenhao Gu, Xiaodong Shen, Weichao Dong, Yuan Dong, Zilong Yang, Laurence T. Computer Vision and Pattern Recognition Language plays a vital role in the realm of human motion. Existing methods have largely depended on CLIP text embeddings for motion generation, yet they fall short in effectively aligning language and motion due to CLIP's pretraining on static image-text pairs. This work introduces LaMP, a novel Language-Motion Pretraining model, which transitions from a language-vision to a more suitable language-motion latent space. It addresses key limitations by generating motion-informative text embeddings, significantly enhancing the relevance and semantics of generated motion sequences. With LaMP, we advance three key tasks: text-to-motion generation, motion-text retrieval, and motion captioning through aligned language-motion representation learning. For generation, we utilize LaMP to provide the text condition instead of CLIP, and an autoregressive masked prediction is designed to achieve mask modeling without rank collapse in transformers. For retrieval, motion features from LaMP's motion transformer interact with query tokens to retrieve text features from the text transformer, and vice versa. For captioning, we finetune a large language model with the language-informative motion features to develop a strong motion captioning model. In addition, we introduce the LaMP-BertScore metric to assess the alignment of generated motions with textual descriptions. Extensive experimental results on multiple datasets demonstrate substantial improvements over previous methods across all three tasks. The code of our method will be made public. |
| title | LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.07093 |