SqueezeMe: Mobile-Ready Distillation of Gaussian Full-Body Avatars
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918074145308672 |
|---|---|
| author | Iandola, Forrest Pidhorskyi, Stanislav Santesteban, Igor Gupta, Divam Pahuja, Anuj Bartolovic, Nemanja Yu, Frank Garbin, Emanuel Simon, Tomas Saito, Shunsuke |
| author_facet | Iandola, Forrest Pidhorskyi, Stanislav Santesteban, Igor Gupta, Divam Pahuja, Anuj Bartolovic, Nemanja Yu, Frank Garbin, Emanuel Simon, Tomas Saito, Shunsuke |
| contents | Gaussian-based human avatars have achieved an unprecedented level of visual fidelity. However, existing approaches based on high-capacity neural networks typically require a desktop GPU to achieve real-time performance for a single avatar, and it remains non-trivial to animate and render such avatars on mobile devices including a standalone VR headset due to substantially limited memory and computational bandwidth. In this paper, we present SqueezeMe, a simple and highly effective framework to convert high-fidelity 3D Gaussian full-body avatars into a lightweight representation that supports both animation and rendering with mobile-grade compute. Our key observation is that the decoding of pose-dependent Gaussian attributes from a neural network creates non-negligible memory and computational overhead. Inspired by blendshapes and linear pose correctives widely used in Computer Graphics, we address this by distilling the pose correctives learned with neural networks into linear layers. Moreover, we further reduce the parameters by sharing the correctives among nearby Gaussians. Combining them with a custom splatting pipeline based on Vulkan, we achieve, for the first time, simultaneous animation and rendering of 3 Gaussian avatars in real-time (72 FPS) on a Meta Quest 3 VR headset. Demo videos are available at https://forresti.github.io/squeezeme. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_15171 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SqueezeMe: Mobile-Ready Distillation of Gaussian Full-Body Avatars Iandola, Forrest Pidhorskyi, Stanislav Santesteban, Igor Gupta, Divam Pahuja, Anuj Bartolovic, Nemanja Yu, Frank Garbin, Emanuel Simon, Tomas Saito, Shunsuke Computer Vision and Pattern Recognition Gaussian-based human avatars have achieved an unprecedented level of visual fidelity. However, existing approaches based on high-capacity neural networks typically require a desktop GPU to achieve real-time performance for a single avatar, and it remains non-trivial to animate and render such avatars on mobile devices including a standalone VR headset due to substantially limited memory and computational bandwidth. In this paper, we present SqueezeMe, a simple and highly effective framework to convert high-fidelity 3D Gaussian full-body avatars into a lightweight representation that supports both animation and rendering with mobile-grade compute. Our key observation is that the decoding of pose-dependent Gaussian attributes from a neural network creates non-negligible memory and computational overhead. Inspired by blendshapes and linear pose correctives widely used in Computer Graphics, we address this by distilling the pose correctives learned with neural networks into linear layers. Moreover, we further reduce the parameters by sharing the correctives among nearby Gaussians. Combining them with a custom splatting pipeline based on Vulkan, we achieve, for the first time, simultaneous animation and rendering of 3 Gaussian avatars in real-time (72 FPS) on a Meta Quest 3 VR headset. Demo videos are available at https://forresti.github.io/squeezeme. |
| title | SqueezeMe: Mobile-Ready Distillation of Gaussian Full-Body Avatars |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.15171 |