MobilePortrait: Real-Time One-Shot Neural Head Avatars on Mobile Devices
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917980227502080 |
|---|---|
| author | Jiang, Jianwen Lin, Gaojie Rong, Zhengkun Liang, Chao Zhu, Yongming Yang, Jiaqi Zhong, Tianyun |
| author_facet | Jiang, Jianwen Lin, Gaojie Rong, Zhengkun Liang, Chao Zhu, Yongming Yang, Jiaqi Zhong, Tianyun |
| contents | Existing neural head avatars methods have achieved significant progress in the image quality and motion range of portrait animation. However, these methods neglect the computational overhead, and to the best of our knowledge, none is designed to run on mobile devices. This paper presents MobilePortrait, a lightweight one-shot neural head avatars method that reduces learning complexity by integrating external knowledge into both the motion modeling and image synthesis, enabling real-time inference on mobile devices. Specifically, we introduce a mixed representation of explicit and implicit keypoints for precise motion modeling and precomputed visual features for enhanced foreground and background synthesis. With these two key designs and using simple U-Nets as backbones, our method achieves state-of-the-art performance with less than one-tenth the computational demand. It has been validated to reach speeds of over 100 FPS on mobile devices and support both video and audio-driven inputs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_05712 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MobilePortrait: Real-Time One-Shot Neural Head Avatars on Mobile Devices Jiang, Jianwen Lin, Gaojie Rong, Zhengkun Liang, Chao Zhu, Yongming Yang, Jiaqi Zhong, Tianyun Computer Vision and Pattern Recognition Artificial Intelligence Existing neural head avatars methods have achieved significant progress in the image quality and motion range of portrait animation. However, these methods neglect the computational overhead, and to the best of our knowledge, none is designed to run on mobile devices. This paper presents MobilePortrait, a lightweight one-shot neural head avatars method that reduces learning complexity by integrating external knowledge into both the motion modeling and image synthesis, enabling real-time inference on mobile devices. Specifically, we introduce a mixed representation of explicit and implicit keypoints for precise motion modeling and precomputed visual features for enhanced foreground and background synthesis. With these two key designs and using simple U-Nets as backbones, our method achieves state-of-the-art performance with less than one-tenth the computational demand. It has been validated to reach speeds of over 100 FPS on mobile devices and support both video and audio-driven inputs. |
| title | MobilePortrait: Real-Time One-Shot Neural Head Avatars on Mobile Devices |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2407.05712 |