TouchScribe: Augmenting Non-Visual Hand-Object Interactions with Automated Live Visual Descriptions
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918327284137984 |
|---|---|
| author | Chang, Ruei-Che Natalie, Rosiana Xu, Wenqian Yap, Jovan Zheng Feng Luo, Tiange Potluri, Venkatesh Guo, Anhong |
| author_facet | Chang, Ruei-Che Natalie, Rosiana Xu, Wenqian Yap, Jovan Zheng Feng Luo, Tiange Potluri, Venkatesh Guo, Anhong |
| contents | People who are blind or have low vision regularly use their hands to interact with the physical world to gain access to objects' shape, size, weight, and texture. However, many rich visual features remain inaccessible through touch alone, making it difficult to distinguish similar objects, interpret visual affordances, and form a complete understanding of objects. In this work, we present TouchScribe, a system that augments hand-object interactions with automated live visual descriptions. We trained a custom egocentric hand interaction model to recognize both common gestures (e.g., grab to inspect, hold side-by-side to compare) and unique ones by blind people (e.g., point to explore color, or swipe to read available texts). Furthermore, TouchScribe provides real-time and adaptive feedback based on hand movement, from hand interaction states, to object labels, and to visual details. Our user study and technical evaluations demonstrate that TouchScribe can provide rich and useful descriptions to support object understanding. Finally, we discuss the implications of making live visual descriptions responsive to users' physical reach. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_07802 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | TouchScribe: Augmenting Non-Visual Hand-Object Interactions with Automated Live Visual Descriptions Chang, Ruei-Che Natalie, Rosiana Xu, Wenqian Yap, Jovan Zheng Feng Luo, Tiange Potluri, Venkatesh Guo, Anhong Human-Computer Interaction People who are blind or have low vision regularly use their hands to interact with the physical world to gain access to objects' shape, size, weight, and texture. However, many rich visual features remain inaccessible through touch alone, making it difficult to distinguish similar objects, interpret visual affordances, and form a complete understanding of objects. In this work, we present TouchScribe, a system that augments hand-object interactions with automated live visual descriptions. We trained a custom egocentric hand interaction model to recognize both common gestures (e.g., grab to inspect, hold side-by-side to compare) and unique ones by blind people (e.g., point to explore color, or swipe to read available texts). Furthermore, TouchScribe provides real-time and adaptive feedback based on hand movement, from hand interaction states, to object labels, and to visual details. Our user study and technical evaluations demonstrate that TouchScribe can provide rich and useful descriptions to support object understanding. Finally, we discuss the implications of making live visual descriptions responsive to users' physical reach. |
| title | TouchScribe: Augmenting Non-Visual Hand-Object Interactions with Automated Live Visual Descriptions |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2602.07802 |