Guided Reality: Generating Visually-Enriched AR Task Guidance with LLMs and Vision Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918146971009024 |
|---|---|
| author | Zhao, Ada Yi Gunturu, Aditya Do, Ellen Yi-Luen Suzuki, Ryo |
| author_facet | Zhao, Ada Yi Gunturu, Aditya Do, Ellen Yi-Luen Suzuki, Ryo |
| contents | Large language models (LLMs) have enabled the automatic generation of step-by-step augmented reality (AR) instructions for a wide range of physical tasks. However, existing LLM-based AR guidance often lacks rich visual augmentations to effectively embed instructions into spatial context for a better user understanding. We present Guided Reality, a fully automated AR system that generates embedded and dynamic visual guidance based on step-by-step instructions. Our system integrates LLMs and vision models to: 1) generate multi-step instructions from user queries, 2) identify appropriate types of visual guidance, 3) extract spatial information about key interaction points in the real world, and 4) embed visual guidance in physical space to support task execution. Drawing from a corpus of user manuals, we define five categories of visual guidance and propose an identification strategy based on the current step. We evaluate the system through a user study (N=16), completing real-world tasks and exploring the system in the wild. Additionally, four instructors shared insights on how Guided Reality could be integrated into their training workflows. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_03547 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Guided Reality: Generating Visually-Enriched AR Task Guidance with LLMs and Vision Models Zhao, Ada Yi Gunturu, Aditya Do, Ellen Yi-Luen Suzuki, Ryo Human-Computer Interaction Large language models (LLMs) have enabled the automatic generation of step-by-step augmented reality (AR) instructions for a wide range of physical tasks. However, existing LLM-based AR guidance often lacks rich visual augmentations to effectively embed instructions into spatial context for a better user understanding. We present Guided Reality, a fully automated AR system that generates embedded and dynamic visual guidance based on step-by-step instructions. Our system integrates LLMs and vision models to: 1) generate multi-step instructions from user queries, 2) identify appropriate types of visual guidance, 3) extract spatial information about key interaction points in the real world, and 4) embed visual guidance in physical space to support task execution. Drawing from a corpus of user manuals, we define five categories of visual guidance and propose an identification strategy based on the current step. We evaluate the system through a user study (N=16), completing real-world tasks and exploring the system in the wild. Additionally, four instructors shared insights on how Guided Reality could be integrated into their training workflows. |
| title | Guided Reality: Generating Visually-Enriched AR Task Guidance with LLMs and Vision Models |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2508.03547 |