Exploring the use of Generative AI to Support Automated Just-in-Time Programming for Visual Scene Displays
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917754578141184 |
|---|---|
| author | Zastudil, Cynthia Holyfield, Christine Kapp, Christine Crosland, Xandria Lorah, Elizabeth Zimmerman, Tara MacNeil, Stephen |
| author_facet | Zastudil, Cynthia Holyfield, Christine Kapp, Christine Crosland, Xandria Lorah, Elizabeth Zimmerman, Tara MacNeil, Stephen |
| contents | Millions of people worldwide rely on alternative and augmentative communication devices to communicate. Visual scene displays (VSDs) can enhance communication for these individuals by embedding communication options within contextualized images. However, existing VSDs often present default images that may lack relevance or require manual configuration, placing a significant burden on communication partners. In this study, we assess the feasibility of leveraging large multimodal models (LMM), such as GPT-4V, to automatically create communication options for VSDs. Communication options were sourced from a LMM and speech-language pathologists (SLPs) and AAC researchers (N=13) for evaluation through an expert assessment conducted by the SLPs and AAC researchers. We present the study's findings, supplemented by insights from semi-structured interviews (N=5) about SLP's and AAC researchers' opinions on the use of generative AI in augmentative and alternative communication devices. Our results indicate that the communication options generated by the LMM were contextually relevant and often resembled those created by humans. However, vital questions remain that must be addressed before LMMs can be confidently implemented in AAC devices. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_11137 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Exploring the use of Generative AI to Support Automated Just-in-Time Programming for Visual Scene Displays Zastudil, Cynthia Holyfield, Christine Kapp, Christine Crosland, Xandria Lorah, Elizabeth Zimmerman, Tara MacNeil, Stephen Human-Computer Interaction Millions of people worldwide rely on alternative and augmentative communication devices to communicate. Visual scene displays (VSDs) can enhance communication for these individuals by embedding communication options within contextualized images. However, existing VSDs often present default images that may lack relevance or require manual configuration, placing a significant burden on communication partners. In this study, we assess the feasibility of leveraging large multimodal models (LMM), such as GPT-4V, to automatically create communication options for VSDs. Communication options were sourced from a LMM and speech-language pathologists (SLPs) and AAC researchers (N=13) for evaluation through an expert assessment conducted by the SLPs and AAC researchers. We present the study's findings, supplemented by insights from semi-structured interviews (N=5) about SLP's and AAC researchers' opinions on the use of generative AI in augmentative and alternative communication devices. Our results indicate that the communication options generated by the LMM were contextually relevant and often resembled those created by humans. However, vital questions remain that must be addressed before LMMs can be confidently implemented in AAC devices. |
| title | Exploring the use of Generative AI to Support Automated Just-in-Time Programming for Visual Scene Displays |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2408.11137 |