Autonomous Workflow for Multimodal Fine-Grained Training Assistants Towards Mixed Reality
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910474209067008 |
|---|---|
| author | Pei, Jiahuan Viola, Irene Huang, Haochen Wang, Junxiao Ahsan, Moonisa Ye, Fanghua Yiming, Jiang Sai, Yao Wang, Di Chen, Zhumin Ren, Pengjie Cesar, Pablo |
| author_facet | Pei, Jiahuan Viola, Irene Huang, Haochen Wang, Junxiao Ahsan, Moonisa Ye, Fanghua Yiming, Jiang Sai, Yao Wang, Di Chen, Zhumin Ren, Pengjie Cesar, Pablo |
| contents | Autonomous artificial intelligence (AI) agents have emerged as promising protocols for automatically understanding the language-based environment, particularly with the exponential development of large language models (LLMs). However, a fine-grained, comprehensive understanding of multimodal environments remains under-explored. This work designs an autonomous workflow tailored for integrating AI agents seamlessly into extended reality (XR) applications for fine-grained training. We present a demonstration of a multimodal fine-grained training assistant for LEGO brick assembly in a pilot XR environment. Specifically, we design a cerebral language agent that integrates LLM with memory, planning, and interaction with XR tools and a vision-language agent, enabling agents to decide their actions based on past experiences. Furthermore, we introduce LEGO-MRTA, a multimodal fine-grained assembly dialogue dataset synthesized automatically in the workflow served by a commercial LLM. This dataset comprises multimodal instruction manuals, conversations, XR responses, and vision question answering. Last, we present several prevailing open-resource LLMs as benchmarks, assessing their performance with and without fine-tuning on the proposed dataset. We anticipate that the broader impact of this workflow will advance the development of smarter assistants for seamless user interaction in XR environments, fostering research in both AI and HCI communities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_13034 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Autonomous Workflow for Multimodal Fine-Grained Training Assistants Towards Mixed Reality Pei, Jiahuan Viola, Irene Huang, Haochen Wang, Junxiao Ahsan, Moonisa Ye, Fanghua Yiming, Jiang Sai, Yao Wang, Di Chen, Zhumin Ren, Pengjie Cesar, Pablo Computation and Language Artificial Intelligence Human-Computer Interaction Autonomous artificial intelligence (AI) agents have emerged as promising protocols for automatically understanding the language-based environment, particularly with the exponential development of large language models (LLMs). However, a fine-grained, comprehensive understanding of multimodal environments remains under-explored. This work designs an autonomous workflow tailored for integrating AI agents seamlessly into extended reality (XR) applications for fine-grained training. We present a demonstration of a multimodal fine-grained training assistant for LEGO brick assembly in a pilot XR environment. Specifically, we design a cerebral language agent that integrates LLM with memory, planning, and interaction with XR tools and a vision-language agent, enabling agents to decide their actions based on past experiences. Furthermore, we introduce LEGO-MRTA, a multimodal fine-grained assembly dialogue dataset synthesized automatically in the workflow served by a commercial LLM. This dataset comprises multimodal instruction manuals, conversations, XR responses, and vision question answering. Last, we present several prevailing open-resource LLMs as benchmarks, assessing their performance with and without fine-tuning on the proposed dataset. We anticipate that the broader impact of this workflow will advance the development of smarter assistants for seamless user interaction in XR environments, fostering research in both AI and HCI communities. |
| title | Autonomous Workflow for Multimodal Fine-Grained Training Assistants Towards Mixed Reality |
| topic | Computation and Language Artificial Intelligence Human-Computer Interaction |
| url | https://arxiv.org/abs/2405.13034 |