Near-realtime Facial Animation by Deep 3D Simulation Super-Resolution
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912033881980928 |
|---|---|
| author | Park, Hyojoon Srinivasan, Sangeetha Grama Cong, Matthew Kim, Doyub Kim, Byungsoo Swartz, Jonathan Museth, Ken Sifakis, Eftychios |
| author_facet | Park, Hyojoon Srinivasan, Sangeetha Grama Cong, Matthew Kim, Doyub Kim, Byungsoo Swartz, Jonathan Museth, Ken Sifakis, Eftychios |
| contents | We present a neural network-based simulation super-resolution framework that can efficiently and realistically enhance a facial performance produced by a low-cost, realtime physics-based simulation to a level of detail that closely approximates that of a reference-quality off-line simulator with much higher resolution (26x element count in our examples) and accurate physical modeling. Our approach is rooted in our ability to construct - via simulation - a training set of paired frames, from the low- and high-resolution simulators respectively, that are in semantic correspondence with each other. We use face animation as an exemplar of such a simulation domain, where creating this semantic congruence is achieved by simply dialing in the same muscle actuation controls and skeletal pose in the two simulators. Our proposed neural network super-resolution framework generalizes from this training set to unseen expressions, compensates for modeling discrepancies between the two simulations due to limited resolution or cost-cutting approximations in the real-time variant, and does not require any semantic descriptors or parameters to be provided as input, other than the result of the real-time simulation. We evaluate the efficacy of our pipeline on a variety of expressive performances and provide comparisons and ablation experiments for plausible variations and alternatives to our proposed scheme. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_03216 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Near-realtime Facial Animation by Deep 3D Simulation Super-Resolution Park, Hyojoon Srinivasan, Sangeetha Grama Cong, Matthew Kim, Doyub Kim, Byungsoo Swartz, Jonathan Museth, Ken Sifakis, Eftychios Graphics Artificial Intelligence We present a neural network-based simulation super-resolution framework that can efficiently and realistically enhance a facial performance produced by a low-cost, realtime physics-based simulation to a level of detail that closely approximates that of a reference-quality off-line simulator with much higher resolution (26x element count in our examples) and accurate physical modeling. Our approach is rooted in our ability to construct - via simulation - a training set of paired frames, from the low- and high-resolution simulators respectively, that are in semantic correspondence with each other. We use face animation as an exemplar of such a simulation domain, where creating this semantic congruence is achieved by simply dialing in the same muscle actuation controls and skeletal pose in the two simulators. Our proposed neural network super-resolution framework generalizes from this training set to unseen expressions, compensates for modeling discrepancies between the two simulations due to limited resolution or cost-cutting approximations in the real-time variant, and does not require any semantic descriptors or parameters to be provided as input, other than the result of the real-time simulation. We evaluate the efficacy of our pipeline on a variety of expressive performances and provide comparisons and ablation experiments for plausible variations and alternatives to our proposed scheme. |
| title | Near-realtime Facial Animation by Deep 3D Simulation Super-Resolution |
| topic | Graphics Artificial Intelligence |
| url | https://arxiv.org/abs/2305.03216 |