Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.04250 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913722041106432 |
|---|---|
| author | Huang, Yifei Xu, Jilan Pei, Baoqi He, Yuping Chen, Guo Zhang, Mingfang Yang, Lijin Nie, Zheng Liu, Jinyao Fan, Guoshun Lin, Dechen Fang, Fang Li, Kunpeng Yuan, Chang Chen, Xinyuan Wang, Yaohui Wang, Yali Qiao, Yu Wang, Limin |
| author_facet | Huang, Yifei Xu, Jilan Pei, Baoqi He, Yuping Chen, Guo Zhang, Mingfang Yang, Lijin Nie, Zheng Liu, Jinyao Fan, Guoshun Lin, Dechen Fang, Fang Li, Kunpeng Yuan, Chang Chen, Xinyuan Wang, Yaohui Wang, Yali Qiao, Yu Wang, Limin |
| contents | We present Vinci, a vision-language system designed to provide real-time, comprehensive AI assistance on portable devices. At its core, Vinci leverages EgoVideo-VL, a novel model that integrates an egocentric vision foundation model with a large language model (LLM), enabling advanced functionalities such as scene understanding, temporal grounding, video summarization, and future planning. To enhance its utility, Vinci incorporates a memory module for processing long video streams in real time while retaining contextual history, a generation module for producing visual action demonstrations, and a retrieval module that bridges egocentric and third-person perspectives to provide relevant how-to videos for skill acquisition. Unlike existing systems that often depend on specialized hardware, Vinci is hardware-agnostic, supporting deployment across a wide range of devices, including smartphones and wearable cameras. In our experiments, we first demonstrate the superior performance of EgoVideo-VL on multiple public benchmarks, showcasing its vision-language reasoning and contextual understanding capabilities. We then conduct a series of user studies to evaluate the real-world effectiveness of Vinci, highlighting its adaptability and usability in diverse scenarios. We hope Vinci can establish a new framework for portable, real-time egocentric AI systems, empowering users with contextual and actionable insights. Including the frontend, backend, and models, all codes of Vinci are available at https://github.com/OpenGVLab/vinci. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_04250 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | An Egocentric Vision-Language Model based Portable Real-time Smart Assistant Huang, Yifei Xu, Jilan Pei, Baoqi He, Yuping Chen, Guo Zhang, Mingfang Yang, Lijin Nie, Zheng Liu, Jinyao Fan, Guoshun Lin, Dechen Fang, Fang Li, Kunpeng Yuan, Chang Chen, Xinyuan Wang, Yaohui Wang, Yali Qiao, Yu Wang, Limin Computer Vision and Pattern Recognition Human-Computer Interaction We present Vinci, a vision-language system designed to provide real-time, comprehensive AI assistance on portable devices. At its core, Vinci leverages EgoVideo-VL, a novel model that integrates an egocentric vision foundation model with a large language model (LLM), enabling advanced functionalities such as scene understanding, temporal grounding, video summarization, and future planning. To enhance its utility, Vinci incorporates a memory module for processing long video streams in real time while retaining contextual history, a generation module for producing visual action demonstrations, and a retrieval module that bridges egocentric and third-person perspectives to provide relevant how-to videos for skill acquisition. Unlike existing systems that often depend on specialized hardware, Vinci is hardware-agnostic, supporting deployment across a wide range of devices, including smartphones and wearable cameras. In our experiments, we first demonstrate the superior performance of EgoVideo-VL on multiple public benchmarks, showcasing its vision-language reasoning and contextual understanding capabilities. We then conduct a series of user studies to evaluate the real-world effectiveness of Vinci, highlighting its adaptability and usability in diverse scenarios. We hope Vinci can establish a new framework for portable, real-time egocentric AI systems, empowering users with contextual and actionable insights. Including the frontend, backend, and models, all codes of Vinci are available at https://github.com/OpenGVLab/vinci. |
| title | An Egocentric Vision-Language Model based Portable Real-time Smart Assistant |
| topic | Computer Vision and Pattern Recognition Human-Computer Interaction |
| url | https://arxiv.org/abs/2503.04250 |