STRIDE: Single-video based Temporally Continuous Occlusion-Robust 3D Pose Estimation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913596017999872 |
|---|---|
| author | Lal, Rohit Bachu, Saketh Garg, Yash Dutta, Arindam Ta, Calvin-Khang Raychaudhuri, Dripta S. Cruz, Hannah Dela Asif, M. Salman Roy-Chowdhury, Amit K. |
| author_facet | Lal, Rohit Bachu, Saketh Garg, Yash Dutta, Arindam Ta, Calvin-Khang Raychaudhuri, Dripta S. Cruz, Hannah Dela Asif, M. Salman Roy-Chowdhury, Amit K. |
| contents | The capability to accurately estimate 3D human poses is crucial for diverse fields such as action recognition, gait recognition, and virtual/augmented reality. However, a persistent and significant challenge within this field is the accurate prediction of human poses under conditions of severe occlusion. Traditional image-based estimators struggle with heavy occlusions due to a lack of temporal context, resulting in inconsistent predictions. While video-based models benefit from processing temporal data, they encounter limitations when faced with prolonged occlusions that extend over multiple frames. This challenge arises because these models struggle to generalize beyond their training datasets, and the variety of occlusions is hard to capture in the training data. Addressing these challenges, we propose STRIDE (Single-video based TempoRally contInuous Occlusion-Robust 3D Pose Estimation), a novel Test-Time Training (TTT) approach to fit a human motion prior for each video. This approach specifically handles occlusions that were not encountered during the model's training. By employing STRIDE, we can refine a sequence of noisy initial pose estimates into accurate, temporally coherent poses during test time, effectively overcoming the limitations of prior methods. Our framework demonstrates flexibility by being model-agnostic, allowing us to use any off-the-shelf 3D pose estimation method for improving robustness and temporal consistency. We validate STRIDE's efficacy through comprehensive experiments on challenging datasets like Occluded Human3.6M, Human3.6M, and OCMotion, where it not only outperforms existing single-image and video-based pose estimation models but also showcases superior handling of substantial occlusions, achieving fast, robust, accurate, and temporally consistent 3D pose estimates. Code is made publicly available at https://github.com/take2rohit/stride |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_16221 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | STRIDE: Single-video based Temporally Continuous Occlusion-Robust 3D Pose Estimation Lal, Rohit Bachu, Saketh Garg, Yash Dutta, Arindam Ta, Calvin-Khang Raychaudhuri, Dripta S. Cruz, Hannah Dela Asif, M. Salman Roy-Chowdhury, Amit K. Computer Vision and Pattern Recognition The capability to accurately estimate 3D human poses is crucial for diverse fields such as action recognition, gait recognition, and virtual/augmented reality. However, a persistent and significant challenge within this field is the accurate prediction of human poses under conditions of severe occlusion. Traditional image-based estimators struggle with heavy occlusions due to a lack of temporal context, resulting in inconsistent predictions. While video-based models benefit from processing temporal data, they encounter limitations when faced with prolonged occlusions that extend over multiple frames. This challenge arises because these models struggle to generalize beyond their training datasets, and the variety of occlusions is hard to capture in the training data. Addressing these challenges, we propose STRIDE (Single-video based TempoRally contInuous Occlusion-Robust 3D Pose Estimation), a novel Test-Time Training (TTT) approach to fit a human motion prior for each video. This approach specifically handles occlusions that were not encountered during the model's training. By employing STRIDE, we can refine a sequence of noisy initial pose estimates into accurate, temporally coherent poses during test time, effectively overcoming the limitations of prior methods. Our framework demonstrates flexibility by being model-agnostic, allowing us to use any off-the-shelf 3D pose estimation method for improving robustness and temporal consistency. We validate STRIDE's efficacy through comprehensive experiments on challenging datasets like Occluded Human3.6M, Human3.6M, and OCMotion, where it not only outperforms existing single-image and video-based pose estimation models but also showcases superior handling of substantial occlusions, achieving fast, robust, accurate, and temporally consistent 3D pose estimates. Code is made publicly available at https://github.com/take2rohit/stride |
| title | STRIDE: Single-video based Temporally Continuous Occlusion-Robust 3D Pose Estimation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2312.16221 |