Holoported Characters: Real-time Free-viewpoint Rendering of Humans from Sparse RGB Cameras
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910540566102016 |
|---|---|
| author | Shetty, Ashwath Habermann, Marc Sun, Guoxing Luvizon, Diogo Golyanik, Vladislav Theobalt, Christian |
| author_facet | Shetty, Ashwath Habermann, Marc Sun, Guoxing Luvizon, Diogo Golyanik, Vladislav Theobalt, Christian |
| contents | We present the first approach to render highly realistic free-viewpoint videos of a human actor in general apparel, from sparse multi-view recording to display, in real-time at an unprecedented 4K resolution. At inference, our method only requires four camera views of the moving actor and the respective 3D skeletal pose. It handles actors in wide clothing, and reproduces even fine-scale dynamic detail, e.g. clothing wrinkles, face expressions, and hand gestures. At training time, our learning-based approach expects dense multi-view video and a rigged static surface scan of the actor. Our method comprises three main stages. Stage 1 is a skeleton-driven neural approach for high-quality capture of the detailed dynamic mesh geometry. Stage 2 is a novel solution to create a view-dependent texture using four test-time camera views as input. Finally, stage 3 comprises a new image-based refinement network rendering the final 4K image given the output from the previous stages. Our approach establishes a new benchmark for real-time rendering resolution and quality using sparse input camera views, unlocking possibilities for immersive telepresence. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_07423 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Holoported Characters: Real-time Free-viewpoint Rendering of Humans from Sparse RGB Cameras Shetty, Ashwath Habermann, Marc Sun, Guoxing Luvizon, Diogo Golyanik, Vladislav Theobalt, Christian Computer Vision and Pattern Recognition We present the first approach to render highly realistic free-viewpoint videos of a human actor in general apparel, from sparse multi-view recording to display, in real-time at an unprecedented 4K resolution. At inference, our method only requires four camera views of the moving actor and the respective 3D skeletal pose. It handles actors in wide clothing, and reproduces even fine-scale dynamic detail, e.g. clothing wrinkles, face expressions, and hand gestures. At training time, our learning-based approach expects dense multi-view video and a rigged static surface scan of the actor. Our method comprises three main stages. Stage 1 is a skeleton-driven neural approach for high-quality capture of the detailed dynamic mesh geometry. Stage 2 is a novel solution to create a view-dependent texture using four test-time camera views as input. Finally, stage 3 comprises a new image-based refinement network rendering the final 4K image given the output from the previous stages. Our approach establishes a new benchmark for real-time rendering resolution and quality using sparse input camera views, unlocking possibilities for immersive telepresence. |
| title | Holoported Characters: Real-time Free-viewpoint Rendering of Humans from Sparse RGB Cameras |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2312.07423 |