MoRAG -- Multi-Fusion Retrieval Augmented Generation for Human Motion
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910737114333184 |
|---|---|
| author | Kalakonda, Sai Shashank Maheshwari, Shubh Sarvadevabhatla, Ravi Kiran |
| author_facet | Kalakonda, Sai Shashank Maheshwari, Shubh Sarvadevabhatla, Ravi Kiran |
| contents | We introduce MoRAG, a novel multi-part fusion based retrieval-augmented generation strategy for text-based human motion generation. The method enhances motion diffusion models by leveraging additional knowledge obtained through an improved motion retrieval process. By effectively prompting large language models (LLMs), we address spelling errors and rephrasing issues in motion retrieval. Our approach utilizes a multi-part retrieval strategy to improve the generalizability of motion retrieval across the language space. We create diverse samples through the spatial composition of the retrieved motions. Furthermore, by utilizing low-level, part-specific motion information, we can construct motion samples for unseen text descriptions. Our experiments demonstrate that our framework can serve as a plug-and-play module, improving the performance of motion diffusion models. Code, pretrained models and sample videos are available at: https://motion-rag.github.io/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_12140 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MoRAG -- Multi-Fusion Retrieval Augmented Generation for Human Motion Kalakonda, Sai Shashank Maheshwari, Shubh Sarvadevabhatla, Ravi Kiran Computer Vision and Pattern Recognition Multimedia We introduce MoRAG, a novel multi-part fusion based retrieval-augmented generation strategy for text-based human motion generation. The method enhances motion diffusion models by leveraging additional knowledge obtained through an improved motion retrieval process. By effectively prompting large language models (LLMs), we address spelling errors and rephrasing issues in motion retrieval. Our approach utilizes a multi-part retrieval strategy to improve the generalizability of motion retrieval across the language space. We create diverse samples through the spatial composition of the retrieved motions. Furthermore, by utilizing low-level, part-specific motion information, we can construct motion samples for unseen text descriptions. Our experiments demonstrate that our framework can serve as a plug-and-play module, improving the performance of motion diffusion models. Code, pretrained models and sample videos are available at: https://motion-rag.github.io/ |
| title | MoRAG -- Multi-Fusion Retrieval Augmented Generation for Human Motion |
| topic | Computer Vision and Pattern Recognition Multimedia |
| url | https://arxiv.org/abs/2409.12140 |