Fish2Mesh Transformer: 3D Human Mesh Recovery from Egocentric Vision
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929748552187904 |
|---|---|
| author | Jeong, David C. Puranik, Aditya Vong, James Deogirikar, Vrushabh Abhijit Fell, Ryan Dietrich, Julianna Kyrarini, Maria Kitts, Christopher |
| author_facet | Jeong, David C. Puranik, Aditya Vong, James Deogirikar, Vrushabh Abhijit Fell, Ryan Dietrich, Julianna Kyrarini, Maria Kitts, Christopher |
| contents | Egocentric human body estimation allows for the inference of user body pose and shape from a wearable camera's first-person perspective. Although research has used pose estimation techniques to overcome self-occlusions and image distortions caused by head-mounted fisheye images, similar advances in 3D human mesh recovery (HMR) techniques have been limited. We introduce Fish2Mesh, a fisheye-aware transformer-based model designed for 3D egocentric human mesh recovery. We propose an egocentric position embedding block to generate an ego-specific position table for the Swin Transformer to reduce fisheye image distortion. Our model utilizes multi-task heads for SMPL parametric regression and camera translations, estimating 3D and 2D joints as auxiliary loss to support model training. To address the scarcity of egocentric camera data, we create a training dataset by employing the pre-trained 4D-Human model and third-person cameras for weak supervision. Our experiments demonstrate that Fish2Mesh outperforms previous state-of-the-art 3D HMR models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_06089 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Fish2Mesh Transformer: 3D Human Mesh Recovery from Egocentric Vision Jeong, David C. Puranik, Aditya Vong, James Deogirikar, Vrushabh Abhijit Fell, Ryan Dietrich, Julianna Kyrarini, Maria Kitts, Christopher Computer Vision and Pattern Recognition Robotics Egocentric human body estimation allows for the inference of user body pose and shape from a wearable camera's first-person perspective. Although research has used pose estimation techniques to overcome self-occlusions and image distortions caused by head-mounted fisheye images, similar advances in 3D human mesh recovery (HMR) techniques have been limited. We introduce Fish2Mesh, a fisheye-aware transformer-based model designed for 3D egocentric human mesh recovery. We propose an egocentric position embedding block to generate an ego-specific position table for the Swin Transformer to reduce fisheye image distortion. Our model utilizes multi-task heads for SMPL parametric regression and camera translations, estimating 3D and 2D joints as auxiliary loss to support model training. To address the scarcity of egocentric camera data, we create a training dataset by employing the pre-trained 4D-Human model and third-person cameras for weak supervision. Our experiments demonstrate that Fish2Mesh outperforms previous state-of-the-art 3D HMR models. |
| title | Fish2Mesh Transformer: 3D Human Mesh Recovery from Egocentric Vision |
| topic | Computer Vision and Pattern Recognition Robotics |
| url | https://arxiv.org/abs/2503.06089 |