Design and Implementation of the Transparent, Interpretable, and Multimodal (TIM) AR Personal Assistant
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908298904600576 |
|---|---|
| author | McGowan, Erin Rulff, Joao Castelo, Sonia Wu, Guande Chen, Shaoyu Lopez, Roque Steers, Bea Roman, Iran R. Dias, Fabio F. Qian, Jing Solunke, Parikshit Middleton, Michael McKendrick, Ryan Silva, Claudio T. |
| author_facet | McGowan, Erin Rulff, Joao Castelo, Sonia Wu, Guande Chen, Shaoyu Lopez, Roque Steers, Bea Roman, Iran R. Dias, Fabio F. Qian, Jing Solunke, Parikshit Middleton, Michael McKendrick, Ryan Silva, Claudio T. |
| contents | The concept of an AI assistant for task guidance is rapidly shifting from a science fiction staple to an impending reality. Such a system is inherently complex, requiring models for perceptual grounding, attention, and reasoning, an intuitive interface that adapts to the performer's needs, and the orchestration of data streams from many sensors. Moreover, all data acquired by the system must be readily available for post-hoc analysis to enable developers to understand performer behavior and quickly detect failures. We introduce TIM, the first end-to-end AI-enabled task guidance system in augmented reality which is capable of detecting both the user and scene as well as providing adaptable, just-in-time feedback. We discuss the system challenges and propose design solutions. We also demonstrate how TIM adapts to domain applications with varying needs, highlighting how the system components can be customized for each scenario. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_02197 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Design and Implementation of the Transparent, Interpretable, and Multimodal (TIM) AR Personal Assistant McGowan, Erin Rulff, Joao Castelo, Sonia Wu, Guande Chen, Shaoyu Lopez, Roque Steers, Bea Roman, Iran R. Dias, Fabio F. Qian, Jing Solunke, Parikshit Middleton, Michael McKendrick, Ryan Silva, Claudio T. Emerging Technologies Human-Computer Interaction The concept of an AI assistant for task guidance is rapidly shifting from a science fiction staple to an impending reality. Such a system is inherently complex, requiring models for perceptual grounding, attention, and reasoning, an intuitive interface that adapts to the performer's needs, and the orchestration of data streams from many sensors. Moreover, all data acquired by the system must be readily available for post-hoc analysis to enable developers to understand performer behavior and quickly detect failures. We introduce TIM, the first end-to-end AI-enabled task guidance system in augmented reality which is capable of detecting both the user and scene as well as providing adaptable, just-in-time feedback. We discuss the system challenges and propose design solutions. We also demonstrate how TIM adapts to domain applications with varying needs, highlighting how the system components can be customized for each scenario. |
| title | Design and Implementation of the Transparent, Interpretable, and Multimodal (TIM) AR Personal Assistant |
| topic | Emerging Technologies Human-Computer Interaction |
| url | https://arxiv.org/abs/2504.02197 |