An Interaction Design Toolkit for Physical Task Guidance with Artificial Intelligence and Mixed Reality
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916707764797440 |
|---|---|
| author | Caetano, Arthur Aponte, Alejandro Sra, Misha |
| author_facet | Caetano, Arthur Aponte, Alejandro Sra, Misha |
| contents | Physical skill acquisition, from sports techniques to surgical procedures, requires instruction and feedback. In the absence of a human expert, Physical Task Guidance (PTG) systems can offer a promising alternative. These systems integrate Artificial Intelligence (AI) and Mixed Reality (MR) to provide realtime feedback and guidance as users practice and learn skills using physical tools and objects. However, designing PTG systems presents challenges beyond engineering complexities. The intricate interplay between users, AI, MR interfaces, and the physical environment creates unique interaction design hurdles. To address these challenges, we present an interaction design toolkit derived from our analysis of PTG prototypes developed by eight student teams during a 10-week-long graduate course. The toolkit comprises Design Considerations, Design Patterns, and an Interaction Canvas. Our evaluation suggests that the toolkit can serve as a valuable resource for practitioners designing PTG systems and researchers developing new tools for human-AI interaction design. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_16892 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | An Interaction Design Toolkit for Physical Task Guidance with Artificial Intelligence and Mixed Reality Caetano, Arthur Aponte, Alejandro Sra, Misha Human-Computer Interaction H.5 Physical skill acquisition, from sports techniques to surgical procedures, requires instruction and feedback. In the absence of a human expert, Physical Task Guidance (PTG) systems can offer a promising alternative. These systems integrate Artificial Intelligence (AI) and Mixed Reality (MR) to provide realtime feedback and guidance as users practice and learn skills using physical tools and objects. However, designing PTG systems presents challenges beyond engineering complexities. The intricate interplay between users, AI, MR interfaces, and the physical environment creates unique interaction design hurdles. To address these challenges, we present an interaction design toolkit derived from our analysis of PTG prototypes developed by eight student teams during a 10-week-long graduate course. The toolkit comprises Design Considerations, Design Patterns, and an Interaction Canvas. Our evaluation suggests that the toolkit can serve as a valuable resource for practitioners designing PTG systems and researchers developing new tools for human-AI interaction design. |
| title | An Interaction Design Toolkit for Physical Task Guidance with Artificial Intelligence and Mixed Reality |
| topic | Human-Computer Interaction H.5 |
| url | https://arxiv.org/abs/2412.16892 |