VimoRAG: Video-based Retrieval-augmented 3D Motion Generation for Motion Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912660577058816 |
|---|---|
| author | Xu, Haidong Xu, Guangwei Zheng, Zhedong Zhu, Xiatian Ji, Wei Li, Xiangtai Guo, Ruijie Zhang, Meishan zhang, Min Fei, Hao |
| author_facet | Xu, Haidong Xu, Guangwei Zheng, Zhedong Zhu, Xiatian Ji, Wei Li, Xiangtai Guo, Ruijie Zhang, Meishan zhang, Min Fei, Hao |
| contents | This paper introduces VimoRAG, a novel video-based retrieval-augmented motion generation framework for motion large language models (LLMs). As motion LLMs face severe out-of-domain/out-of-vocabulary issues due to limited annotated data, VimoRAG leverages large-scale in-the-wild video databases to enhance 3D motion generation by retrieving relevant 2D human motion signals. While video-based motion RAG is nontrivial, we address two key bottlenecks: (1) developing an effective motion-centered video retrieval model that distinguishes human poses and actions, and (2) mitigating the issue of error propagation caused by suboptimal retrieval results. We design the Gemini Motion Video Retriever mechanism and the Motion-centric Dual-alignment DPO Trainer, enabling effective retrieval and generation processes. Experimental results show that VimoRAG significantly boosts the performance of motion LLMs constrained to text-only input. All the resources are available at https://walkermitty.github.io/VimoRAG/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_12081 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VimoRAG: Video-based Retrieval-augmented 3D Motion Generation for Motion Language Models Xu, Haidong Xu, Guangwei Zheng, Zhedong Zhu, Xiatian Ji, Wei Li, Xiangtai Guo, Ruijie Zhang, Meishan zhang, Min Fei, Hao Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language This paper introduces VimoRAG, a novel video-based retrieval-augmented motion generation framework for motion large language models (LLMs). As motion LLMs face severe out-of-domain/out-of-vocabulary issues due to limited annotated data, VimoRAG leverages large-scale in-the-wild video databases to enhance 3D motion generation by retrieving relevant 2D human motion signals. While video-based motion RAG is nontrivial, we address two key bottlenecks: (1) developing an effective motion-centered video retrieval model that distinguishes human poses and actions, and (2) mitigating the issue of error propagation caused by suboptimal retrieval results. We design the Gemini Motion Video Retriever mechanism and the Motion-centric Dual-alignment DPO Trainer, enabling effective retrieval and generation processes. Experimental results show that VimoRAG significantly boosts the performance of motion LLMs constrained to text-only input. All the resources are available at https://walkermitty.github.io/VimoRAG/ |
| title | VimoRAG: Video-based Retrieval-augmented 3D Motion Generation for Motion Language Models |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2508.12081 |