GesPrompt: Leveraging Co-Speech Gestures to Augment LLM-Based Interaction in Virtual Reality
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913827264659456 |
|---|---|
| author | Hu, Xiyun Ma, Dizhi He, Fengming Zhu, Zhengzhe Hsia, Shao-Kang Zhu, Chenfei Liu, Ziyi Ramani, Karthik |
| author_facet | Hu, Xiyun Ma, Dizhi He, Fengming Zhu, Zhengzhe Hsia, Shao-Kang Zhu, Chenfei Liu, Ziyi Ramani, Karthik |
| contents | Large Language Model (LLM)-based copilots have shown great potential in Extended Reality (XR) applications. However, the user faces challenges when describing the 3D environments to the copilots due to the complexity of conveying spatial-temporal information through text or speech alone. To address this, we introduce GesPrompt, a multimodal XR interface that combines co-speech gestures with speech, allowing end-users to communicate more naturally and accurately with LLM-based copilots in XR environments. By incorporating gestures, GesPrompt extracts spatial-temporal reference from co-speech gestures, reducing the need for precise textual prompts and minimizing cognitive load for end-users. Our contributions include (1) a workflow to integrate gesture and speech input in the XR environment, (2) a prototype VR system that implements the workflow, and (3) a user study demonstrating its effectiveness in improving user communication in VR environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_05441 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GesPrompt: Leveraging Co-Speech Gestures to Augment LLM-Based Interaction in Virtual Reality Hu, Xiyun Ma, Dizhi He, Fengming Zhu, Zhengzhe Hsia, Shao-Kang Zhu, Chenfei Liu, Ziyi Ramani, Karthik Human-Computer Interaction Large Language Model (LLM)-based copilots have shown great potential in Extended Reality (XR) applications. However, the user faces challenges when describing the 3D environments to the copilots due to the complexity of conveying spatial-temporal information through text or speech alone. To address this, we introduce GesPrompt, a multimodal XR interface that combines co-speech gestures with speech, allowing end-users to communicate more naturally and accurately with LLM-based copilots in XR environments. By incorporating gestures, GesPrompt extracts spatial-temporal reference from co-speech gestures, reducing the need for precise textual prompts and minimizing cognitive load for end-users. Our contributions include (1) a workflow to integrate gesture and speech input in the XR environment, (2) a prototype VR system that implements the workflow, and (3) a user study demonstrating its effectiveness in improving user communication in VR environments. |
| title | GesPrompt: Leveraging Co-Speech Gestures to Augment LLM-Based Interaction in Virtual Reality |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2505.05441 |