Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.21080 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909444513726464 |
|---|---|
| author | Huang, Yifei Xu, Jilan Pei, Baoqi He, Yuping Chen, Guo Yang, Lijin Chen, Xinyuan Wang, Yaohui Nie, Zheng Liu, Jinyao Fan, Guoshun Lin, Dechen Fang, Fang Li, Kunpeng Yuan, Chang Wang, Yali Qiao, Yu Wang, Limin |
| author_facet | Huang, Yifei Xu, Jilan Pei, Baoqi He, Yuping Chen, Guo Yang, Lijin Chen, Xinyuan Wang, Yaohui Nie, Zheng Liu, Jinyao Fan, Guoshun Lin, Dechen Fang, Fang Li, Kunpeng Yuan, Chang Wang, Yali Qiao, Yu Wang, Limin |
| contents | We introduce Vinci, a real-time embodied smart assistant built upon an egocentric vision-language model. Designed for deployment on portable devices such as smartphones and wearable cameras, Vinci operates in an "always on" mode, continuously observing the environment to deliver seamless interaction and assistance. Users can wake up the system and engage in natural conversations to ask questions or seek assistance, with responses delivered through audio for hands-free convenience. With its ability to process long video streams in real-time, Vinci can answer user queries about current observations and historical context while also providing task planning based on past interactions. To further enhance usability, Vinci integrates a video generation module that creates step-by-step visual demonstrations for tasks that require detailed guidance. We hope that Vinci can establish a robust framework for portable, real-time egocentric AI systems, empowering users with contextual and actionable insights. We release the complete implementation for the development of the device in conjunction with a demo web platform to test uploaded videos at https://github.com/OpenGVLab/vinci. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_21080 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Model Huang, Yifei Xu, Jilan Pei, Baoqi He, Yuping Chen, Guo Yang, Lijin Chen, Xinyuan Wang, Yaohui Nie, Zheng Liu, Jinyao Fan, Guoshun Lin, Dechen Fang, Fang Li, Kunpeng Yuan, Chang Wang, Yali Qiao, Yu Wang, Limin Computer Vision and Pattern Recognition We introduce Vinci, a real-time embodied smart assistant built upon an egocentric vision-language model. Designed for deployment on portable devices such as smartphones and wearable cameras, Vinci operates in an "always on" mode, continuously observing the environment to deliver seamless interaction and assistance. Users can wake up the system and engage in natural conversations to ask questions or seek assistance, with responses delivered through audio for hands-free convenience. With its ability to process long video streams in real-time, Vinci can answer user queries about current observations and historical context while also providing task planning based on past interactions. To further enhance usability, Vinci integrates a video generation module that creates step-by-step visual demonstrations for tasks that require detailed guidance. We hope that Vinci can establish a robust framework for portable, real-time egocentric AI systems, empowering users with contextual and actionable insights. We release the complete implementation for the development of the device in conjunction with a demo web platform to test uploaded videos at https://github.com/OpenGVLab/vinci. |
| title | Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Model |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.21080 |