LightAvatar: Efficient Head Avatar as Dynamic Neural Light Field

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Wang, Huan, Tan, Feitong, Bai, Ziqian, Zhang, Yinda, Liu, Shichen, Xu, Qiangeng, Chai, Menglei, Prabhu, Anish, Pandey, Rohit, Fanello, Sean, Huang, Zeng, Fu, Yun
Format:	Preprint
Published:	2024
Subjects:	Computer Vision and Pattern Recognition
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

_version_	1866915008040927232
author	Wang, Huan Tan, Feitong Bai, Ziqian Zhang, Yinda Liu, Shichen Xu, Qiangeng Chai, Menglei Prabhu, Anish Pandey, Rohit Fanello, Sean Huang, Zeng Fu, Yun
author_facet	Wang, Huan Tan, Feitong Bai, Ziqian Zhang, Yinda Liu, Shichen Xu, Qiangeng Chai, Menglei Prabhu, Anish Pandey, Rohit Fanello, Sean Huang, Zeng Fu, Yun
contents	Recent works have shown that neural radiance fields (NeRFs) on top of parametric models have reached SOTA quality to build photorealistic head avatars from a monocular video. However, one major limitation of the NeRF-based avatars is the slow rendering speed due to the dense point sampling of NeRF, preventing them from broader utility on resource-constrained devices. We introduce LightAvatar, the first head avatar model based on neural light fields (NeLFs). LightAvatar renders an image from 3DMM parameters and a camera pose via a single network forward pass, without using mesh or volume rendering. The proposed approach, while being conceptually appealing, poses a significant challenge towards real-time efficiency and training stability. To resolve them, we introduce dedicated network designs to obtain proper representations for the NeLF model and maintain a low FLOPs budget. Meanwhile, we tap into a distillation-based training strategy that uses a pretrained avatar model as teacher to synthesize abundant pseudo data for training. A warping field network is introduced to correct the fitting error in the real data so that the model can learn better. Extensive experiments suggest that our method can achieve new SOTA image quality quantitatively or qualitatively, while being significantly faster than the counterparts, reporting 174.1 FPS (512x512 resolution) on a consumer-grade GPU (RTX3090) with no customized optimization.
format	Preprint
id	arxiv_https___arxiv_org_abs_2409_18057
institution	arXiv
publishDate	2024
record_format	arxiv
spellingShingle	LightAvatar: Efficient Head Avatar as Dynamic Neural Light Field Wang, Huan Tan, Feitong Bai, Ziqian Zhang, Yinda Liu, Shichen Xu, Qiangeng Chai, Menglei Prabhu, Anish Pandey, Rohit Fanello, Sean Huang, Zeng Fu, Yun Computer Vision and Pattern Recognition Recent works have shown that neural radiance fields (NeRFs) on top of parametric models have reached SOTA quality to build photorealistic head avatars from a monocular video. However, one major limitation of the NeRF-based avatars is the slow rendering speed due to the dense point sampling of NeRF, preventing them from broader utility on resource-constrained devices. We introduce LightAvatar, the first head avatar model based on neural light fields (NeLFs). LightAvatar renders an image from 3DMM parameters and a camera pose via a single network forward pass, without using mesh or volume rendering. The proposed approach, while being conceptually appealing, poses a significant challenge towards real-time efficiency and training stability. To resolve them, we introduce dedicated network designs to obtain proper representations for the NeLF model and maintain a low FLOPs budget. Meanwhile, we tap into a distillation-based training strategy that uses a pretrained avatar model as teacher to synthesize abundant pseudo data for training. A warping field network is introduced to correct the fitting error in the real data so that the model can learn better. Extensive experiments suggest that our method can achieve new SOTA image quality quantitatively or qualitatively, while being significantly faster than the counterparts, reporting 174.1 FPS (512x512 resolution) on a consumer-grade GPU (RTX3090) with no customized optimization.
title	LightAvatar: Efficient Head Avatar as Dynamic Neural Light Field
topic	Computer Vision and Pattern Recognition
url	https://arxiv.org/abs/2409.18057