VOICE: Visual Oracle for Interaction, Conversation, and Explanation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909052998516736 |
|---|---|
| author | Jia, Donggang Irger, Alexandra Besancon, Lonni Strnad, Ondrej Luo, Deng Bjorklund, Johanna Ynnerman, Anders Viola, Ivan |
| author_facet | Jia, Donggang Irger, Alexandra Besancon, Lonni Strnad, Ondrej Luo, Deng Bjorklund, Johanna Ynnerman, Anders Viola, Ivan |
| contents | We present VOICE, a novel approach to science communication that connects large language models' (LLM) conversational capabilities with interactive exploratory visualization. VOICE introduces several innovative technical contributions that drive our conversational visualization framework. Our foundation is a pack-of-bots that can perform specific tasks, such as assigning tasks, extracting instructions, and generating coherent content. We employ fine-tuning and prompt engineering techniques to tailor bots' performance to their specific roles and accurately respond to user queries. Our interactive text-to-visualization method generates a flythrough sequence matching the content explanation. Besides, natural language interaction provides capabilities to navigate and manipulate the 3D models in real-time. The VOICE framework can receive arbitrary voice commands from the user and respond verbally, tightly coupled with corresponding visual representation with low latency and high accuracy. We demonstrate the effectiveness of our approach by applying it to the molecular visualization domain: analyzing three 3D molecular models with multi-scale and multi-instance attributes. We finally evaluate VOICE with the identified educational experts to show the potential of our approach. All supplemental materials are available at https://osf.io/g7fbr. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2304_04083 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | VOICE: Visual Oracle for Interaction, Conversation, and Explanation Jia, Donggang Irger, Alexandra Besancon, Lonni Strnad, Ondrej Luo, Deng Bjorklund, Johanna Ynnerman, Anders Viola, Ivan Human-Computer Interaction Graphics We present VOICE, a novel approach to science communication that connects large language models' (LLM) conversational capabilities with interactive exploratory visualization. VOICE introduces several innovative technical contributions that drive our conversational visualization framework. Our foundation is a pack-of-bots that can perform specific tasks, such as assigning tasks, extracting instructions, and generating coherent content. We employ fine-tuning and prompt engineering techniques to tailor bots' performance to their specific roles and accurately respond to user queries. Our interactive text-to-visualization method generates a flythrough sequence matching the content explanation. Besides, natural language interaction provides capabilities to navigate and manipulate the 3D models in real-time. The VOICE framework can receive arbitrary voice commands from the user and respond verbally, tightly coupled with corresponding visual representation with low latency and high accuracy. We demonstrate the effectiveness of our approach by applying it to the molecular visualization domain: analyzing three 3D molecular models with multi-scale and multi-instance attributes. We finally evaluate VOICE with the identified educational experts to show the potential of our approach. All supplemental materials are available at https://osf.io/g7fbr. |
| title | VOICE: Visual Oracle for Interaction, Conversation, and Explanation |
| topic | Human-Computer Interaction Graphics |
| url | https://arxiv.org/abs/2304.04083 |