CUIfy the XR: An Open-Source Package to Embed LLM-powered Conversational Agents in XR
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913715851362304 |
|---|---|
| author | Buldu, Kadir Burak Özdel, Süleyman Lau, Ka Hei Carrie Wang, Mengdi Saad, Daniel Schönborn, Sofie Boch, Auxane Kasneci, Enkelejda Bozkir, Efe |
| author_facet | Buldu, Kadir Burak Özdel, Süleyman Lau, Ka Hei Carrie Wang, Mengdi Saad, Daniel Schönborn, Sofie Boch, Auxane Kasneci, Enkelejda Bozkir, Efe |
| contents | Recent developments in computer graphics, machine learning, and sensor technologies enable numerous opportunities for extended reality (XR) setups for everyday life, from skills training to entertainment. With large corporations offering affordable consumer-grade head-mounted displays (HMDs), XR will likely become pervasive, and HMDs will develop as personal devices like smartphones and tablets. However, having intelligent spaces and naturalistic interactions in XR is as important as technological advances so that users grow their engagement in virtual and augmented spaces. To this end, large language model (LLM)--powered non-player characters (NPCs) with speech-to-text (STT) and text-to-speech (TTS) models bring significant advantages over conventional or pre-scripted NPCs for facilitating more natural conversational user interfaces (CUIs) in XR. This paper provides the community with an open-source, customizable, extendable, and privacy-aware Unity package, CUIfy, that facilitates speech-based NPC-user interaction with widely used LLMs, STT, and TTS models. Our package also supports multiple LLM-powered NPCs per environment and minimizes latency between different computational models through streaming to achieve usable interactions between users and NPCs. We publish our source code in the following repository: https://gitlab.lrz.de/hctl/cuify |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_04671 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CUIfy the XR: An Open-Source Package to Embed LLM-powered Conversational Agents in XR Buldu, Kadir Burak Özdel, Süleyman Lau, Ka Hei Carrie Wang, Mengdi Saad, Daniel Schönborn, Sofie Boch, Auxane Kasneci, Enkelejda Bozkir, Efe Human-Computer Interaction Artificial Intelligence Recent developments in computer graphics, machine learning, and sensor technologies enable numerous opportunities for extended reality (XR) setups for everyday life, from skills training to entertainment. With large corporations offering affordable consumer-grade head-mounted displays (HMDs), XR will likely become pervasive, and HMDs will develop as personal devices like smartphones and tablets. However, having intelligent spaces and naturalistic interactions in XR is as important as technological advances so that users grow their engagement in virtual and augmented spaces. To this end, large language model (LLM)--powered non-player characters (NPCs) with speech-to-text (STT) and text-to-speech (TTS) models bring significant advantages over conventional or pre-scripted NPCs for facilitating more natural conversational user interfaces (CUIs) in XR. This paper provides the community with an open-source, customizable, extendable, and privacy-aware Unity package, CUIfy, that facilitates speech-based NPC-user interaction with widely used LLMs, STT, and TTS models. Our package also supports multiple LLM-powered NPCs per environment and minimizes latency between different computational models through streaming to achieve usable interactions between users and NPCs. We publish our source code in the following repository: https://gitlab.lrz.de/hctl/cuify |
| title | CUIfy the XR: An Open-Source Package to Embed LLM-powered Conversational Agents in XR |
| topic | Human-Computer Interaction Artificial Intelligence |
| url | https://arxiv.org/abs/2411.04671 |