RealitySummary: Exploring On-Demand Mixed Reality Text Summarization and Question Answering using Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908555837177856 |
|---|---|
| author | Gunturu, Aditya Jadon, Shivesh Zhang, Nandi Faraji, Morteza Thundathil, Jarin Willett, Wesley Suzuki, Ryo |
| author_facet | Gunturu, Aditya Jadon, Shivesh Zhang, Nandi Faraji, Morteza Thundathil, Jarin Willett, Wesley Suzuki, Ryo |
| contents | Large Language Models (LLMs) are gaining popularity as reading and summarization aids. However, little is known about their potential benefits when integrated with mixed reality (MR) interfaces to support everyday reading. In this iterative investigation, we developed RealitySummary, an MR reading assistant that seamlessly integrates LLMs with always-on camera access, OCR-based text extraction, and augmented spatial and visual responses. Developed iteratively, RealitySummary evolved across three versions, each shaped by user feedback and reflective analysis: 1) a preliminary user study to understand reader perceptions (N=12), 2) an in-the-wild deployment to explore real-world usage (N=11), and 3) a diary study to capture insights from real-world work contexts (N=5). Our empirical studies' findings highlight the unique advantages of combining AI and MR, including always-on implicit assistance, long-term temporal history, minimal context switching, and spatial affordances, demonstrating significant potential for future LLM-MR interfaces beyond traditional screen-based interactions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_18620 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | RealitySummary: Exploring On-Demand Mixed Reality Text Summarization and Question Answering using Large Language Models Gunturu, Aditya Jadon, Shivesh Zhang, Nandi Faraji, Morteza Thundathil, Jarin Willett, Wesley Suzuki, Ryo Human-Computer Interaction Artificial Intelligence Computation and Language Large Language Models (LLMs) are gaining popularity as reading and summarization aids. However, little is known about their potential benefits when integrated with mixed reality (MR) interfaces to support everyday reading. In this iterative investigation, we developed RealitySummary, an MR reading assistant that seamlessly integrates LLMs with always-on camera access, OCR-based text extraction, and augmented spatial and visual responses. Developed iteratively, RealitySummary evolved across three versions, each shaped by user feedback and reflective analysis: 1) a preliminary user study to understand reader perceptions (N=12), 2) an in-the-wild deployment to explore real-world usage (N=11), and 3) a diary study to capture insights from real-world work contexts (N=5). Our empirical studies' findings highlight the unique advantages of combining AI and MR, including always-on implicit assistance, long-term temporal history, minimal context switching, and spatial affordances, demonstrating significant potential for future LLM-MR interfaces beyond traditional screen-based interactions. |
| title | RealitySummary: Exploring On-Demand Mixed Reality Text Summarization and Question Answering using Large Language Models |
| topic | Human-Computer Interaction Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2405.18620 |