Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2312.01964 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911839980355584 |
|---|---|
| author | Zhang, Haodong Chen, ZhiKe Xu, Haocheng Hao, Lei Wu, Xiaofei Xu, Songcen Zhang, Zhensong Wang, Yue Xiong, Rong |
| author_facet | Zhang, Haodong Chen, ZhiKe Xu, Haocheng Hao, Lei Wu, Xiaofei Xu, Songcen Zhang, Zhensong Wang, Yue Xiong, Rong |
| contents | Capturing and preserving motion semantics is essential to motion retargeting between animation characters. However, most of the previous works neglect the semantic information or rely on human-designed joint-level representations. Here, we present a novel Semantics-aware Motion reTargeting (SMT) method with the advantage of vision-language models to extract and maintain meaningful motion semantics. We utilize a differentiable module to render 3D motions. Then the high-level motion semantics are incorporated into the motion retargeting process by feeding the vision-language model with the rendered images and aligning the extracted semantic embeddings. To ensure the preservation of fine-grained motion details and high-level semantics, we adopt a two-stage pipeline consisting of skeleton-aware pre-training and fine-tuning with semantics and geometry constraints. Experimental results show the effectiveness of the proposed method in producing high-quality motion retargeting results while accurately preserving motion semantics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_01964 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Semantics-aware Motion Retargeting with Vision-Language Models Zhang, Haodong Chen, ZhiKe Xu, Haocheng Hao, Lei Wu, Xiaofei Xu, Songcen Zhang, Zhensong Wang, Yue Xiong, Rong Computer Vision and Pattern Recognition Graphics Capturing and preserving motion semantics is essential to motion retargeting between animation characters. However, most of the previous works neglect the semantic information or rely on human-designed joint-level representations. Here, we present a novel Semantics-aware Motion reTargeting (SMT) method with the advantage of vision-language models to extract and maintain meaningful motion semantics. We utilize a differentiable module to render 3D motions. Then the high-level motion semantics are incorporated into the motion retargeting process by feeding the vision-language model with the rendered images and aligning the extracted semantic embeddings. To ensure the preservation of fine-grained motion details and high-level semantics, we adopt a two-stage pipeline consisting of skeleton-aware pre-training and fine-tuning with semantics and geometry constraints. Experimental results show the effectiveness of the proposed method in producing high-quality motion retargeting results while accurately preserving motion semantics. |
| title | Semantics-aware Motion Retargeting with Vision-Language Models |
| topic | Computer Vision and Pattern Recognition Graphics |
| url | https://arxiv.org/abs/2312.01964 |