Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.17368 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909390262501376 |
|---|---|
| author | Peiffer, J. D. Shah, Kunal Anarwala, Shawana Abdou, Kayan Cotton, R. James |
| author_facet | Peiffer, J. D. Shah, Kunal Anarwala, Shawana Abdou, Kayan Cotton, R. James |
| contents | Video and wearable sensor data provide complementary information about human movement. Video provides a holistic understanding of the entire body in the world while wearable sensors provide high-resolution measurements of specific body segments. A robust method to fuse these modalities and obtain biomechanically accurate kinematics would have substantial utility for clinical assessment and monitoring. While multiple video-sensor fusion methods exist, most assume that a time-intensive, and often brittle, sensor-body calibration process has already been performed. In this work, we present a method to combine handheld smartphone video and uncalibrated wearable sensor data at their full temporal resolution. Our monocular, video-only, biomechanical reconstruction already performs well, with only several degrees of error at the knee during walking compared to markerless motion capture. Reconstructing from a fusion of video and wearable sensor data further reduces this error. We validate this in a mixture of people with no gait impairments, lower limb prosthesis users, and individuals with a history of stroke. We also show that sensor data allows tracking through periods of visual occlusion. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_17368 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Fusing uncalibrated IMUs and handheld smartphone video to reconstruct knee kinematics Peiffer, J. D. Shah, Kunal Anarwala, Shawana Abdou, Kayan Cotton, R. James Computer Vision and Pattern Recognition Video and wearable sensor data provide complementary information about human movement. Video provides a holistic understanding of the entire body in the world while wearable sensors provide high-resolution measurements of specific body segments. A robust method to fuse these modalities and obtain biomechanically accurate kinematics would have substantial utility for clinical assessment and monitoring. While multiple video-sensor fusion methods exist, most assume that a time-intensive, and often brittle, sensor-body calibration process has already been performed. In this work, we present a method to combine handheld smartphone video and uncalibrated wearable sensor data at their full temporal resolution. Our monocular, video-only, biomechanical reconstruction already performs well, with only several degrees of error at the knee during walking compared to markerless motion capture. Reconstructing from a fusion of video and wearable sensor data further reduces this error. We validate this in a mixture of people with no gait impairments, lower limb prosthesis users, and individuals with a history of stroke. We also show that sensor data allows tracking through periods of visual occlusion. |
| title | Fusing uncalibrated IMUs and handheld smartphone video to reconstruct knee kinematics |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2405.17368 |