WatchHand: Enabling Continuous Hand Pose Tracking On Off-the-Shelf Smartwatches
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915815845003264 |
|---|---|
| author | Kim, Jiwan Lee, Chi-Jung Jung, Hohurn Yu, Tianhong Catherine Zhang, Ruidong Oakley, Ian Zhang, Cheng |
| author_facet | Kim, Jiwan Lee, Chi-Jung Jung, Hohurn Yu, Tianhong Catherine Zhang, Ruidong Oakley, Ian Zhang, Cheng |
| contents | Tracking hand poses on wrist-wearables enables rich, expressive interactions, yet remains unavailable on commercial smartwatches, as prior implementations rely on external sensors or custom hardware, limiting their real-world applicability. To address this, we present WatchHand, the first continuous 3D hand pose tracking system implemented on off-the-shelf smartwatches using only their built-in speaker and microphone. WatchHand emits inaudible frequency-modulated continuous waves and captures their reflections from the hand. These acoustic signals are processed by a deep-learning model that estimates 3D hand poses for 20 finger joints. We evaluate WatchHand across diverse real-world conditions -- multiple smartwatch models, wearing-hands, body postures, noise conditions, pose-variation protocols -- and achieve a mean per-joint position error of 7.87 mm in cross-session tests with device remounting. Although performance drops for unseen users or gestures, the model adapts effectively with lightweight fine-tuning on small amounts of data. Overall, WatchHand lowers the barrier to smartwatch-based hand tracking by eliminating additional hardware while enabling robust, always-available interactions on millions of existing devices. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_21610 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | WatchHand: Enabling Continuous Hand Pose Tracking On Off-the-Shelf Smartwatches Kim, Jiwan Lee, Chi-Jung Jung, Hohurn Yu, Tianhong Catherine Zhang, Ruidong Oakley, Ian Zhang, Cheng Human-Computer Interaction Tracking hand poses on wrist-wearables enables rich, expressive interactions, yet remains unavailable on commercial smartwatches, as prior implementations rely on external sensors or custom hardware, limiting their real-world applicability. To address this, we present WatchHand, the first continuous 3D hand pose tracking system implemented on off-the-shelf smartwatches using only their built-in speaker and microphone. WatchHand emits inaudible frequency-modulated continuous waves and captures their reflections from the hand. These acoustic signals are processed by a deep-learning model that estimates 3D hand poses for 20 finger joints. We evaluate WatchHand across diverse real-world conditions -- multiple smartwatch models, wearing-hands, body postures, noise conditions, pose-variation protocols -- and achieve a mean per-joint position error of 7.87 mm in cross-session tests with device remounting. Although performance drops for unseen users or gestures, the model adapts effectively with lightweight fine-tuning on small amounts of data. Overall, WatchHand lowers the barrier to smartwatch-based hand tracking by eliminating additional hardware while enabling robust, always-available interactions on millions of existing devices. |
| title | WatchHand: Enabling Continuous Hand Pose Tracking On Off-the-Shelf Smartwatches |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2602.21610 |