Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2402.03944 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909319519272960 |
|---|---|
| author | Wang, Youjia Wu, Yiwen Zhou, Hengan Lin, Hongyang Peng, Xingyue Zhang, Jingyan Zhu, Yingsheng Jiang, Yingwenqi Zhang, Yatu Xu, Lan Wang, Jingya Yu, Jingyi |
| author_facet | Wang, Youjia Wu, Yiwen Zhou, Hengan Lin, Hongyang Peng, Xingyue Zhang, Jingyan Zhu, Yingsheng Jiang, Yingwenqi Zhang, Yatu Xu, Lan Wang, Jingya Yu, Jingyi |
| contents | We present Capturing the Unseen (CAPUS), a novel facial motion capture (MoCap) technique that operates without visual signals. CAPUS leverages miniaturized Inertial Measurement Units (IMUs) as a new sensing modality for facial motion capture. While IMUs have become essential in full-body MoCap for their portability and independence from environmental conditions, their application in facial MoCap remains underexplored. We address this by customizing micro-IMUs, small enough to be placed on the face, and strategically positioning them in alignment with key facial muscles to capture expression dynamics. CAPUS introduces the first facial IMU dataset, encompassing both IMU and visual signals from participants engaged in diverse activities such as multilingual speech, facial expressions, and emotionally intoned auditions. We train a Transformer Diffusion-based neural network to infer Blendshape parameters directly from IMU data. Our experimental results demonstrate that CAPUS reliably captures facial motion in conditions where visual-based methods struggle, including facial occlusions, rapid movements, and low-light environments. Additionally, by eliminating the need for visual inputs, CAPUS offers enhanced privacy protection, making it a robust solution for vision-free facial MoCap. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_03944 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Capturing the Unseen: Vision-Free Facial Motion Capture Using Inertial Measurement Units Wang, Youjia Wu, Yiwen Zhou, Hengan Lin, Hongyang Peng, Xingyue Zhang, Jingyan Zhu, Yingsheng Jiang, Yingwenqi Zhang, Yatu Xu, Lan Wang, Jingya Yu, Jingyi Computer Vision and Pattern Recognition We present Capturing the Unseen (CAPUS), a novel facial motion capture (MoCap) technique that operates without visual signals. CAPUS leverages miniaturized Inertial Measurement Units (IMUs) as a new sensing modality for facial motion capture. While IMUs have become essential in full-body MoCap for their portability and independence from environmental conditions, their application in facial MoCap remains underexplored. We address this by customizing micro-IMUs, small enough to be placed on the face, and strategically positioning them in alignment with key facial muscles to capture expression dynamics. CAPUS introduces the first facial IMU dataset, encompassing both IMU and visual signals from participants engaged in diverse activities such as multilingual speech, facial expressions, and emotionally intoned auditions. We train a Transformer Diffusion-based neural network to infer Blendshape parameters directly from IMU data. Our experimental results demonstrate that CAPUS reliably captures facial motion in conditions where visual-based methods struggle, including facial occlusions, rapid movements, and low-light environments. Additionally, by eliminating the need for visual inputs, CAPUS offers enhanced privacy protection, making it a robust solution for vision-free facial MoCap. |
| title | Capturing the Unseen: Vision-Free Facial Motion Capture Using Inertial Measurement Units |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2402.03944 |