Ego-EXTRA: video-language Egocentric Dataset for EXpert-TRAinee assistance
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915675842281472 |
|---|---|
| author | Ragusa, Francesco Mazzamuto, Michele Forte, Rosario D'Ambra, Irene Fort, James Engel, Jakob Furnari, Antonino Farinella, Giovanni Maria |
| author_facet | Ragusa, Francesco Mazzamuto, Michele Forte, Rosario D'Ambra, Irene Fort, James Engel, Jakob Furnari, Antonino Farinella, Giovanni Maria |
| contents | We present Ego-EXTRA, a video-language Egocentric Dataset for EXpert-TRAinee assistance. Ego-EXTRA features 50 hours of unscripted egocentric videos of subjects performing procedural activities (the trainees) while guided by real-world experts who provide guidance and answer specific questions using natural language. Following a ``Wizard of OZ'' data collection paradigm, the expert enacts a wearable intelligent assistant, looking at the activities performed by the trainee exclusively from their egocentric point of view, answering questions when asked by the trainee, or proactively interacting with suggestions during the procedures. This unique data collection protocol enables Ego-EXTRA to capture a high-quality dialogue in which expert-level feedback is provided to the trainee. Two-way dialogues between experts and trainees are recorded, transcribed, and used to create a novel benchmark comprising more than 15k high-quality Visual Question Answer sets, which we use to evaluate Multimodal Large Language Models. The results show that Ego-EXTRA is challenging and highlight the limitations of current models when used to provide expert-level assistance to the user. The Ego-EXTRA dataset is publicly available to support the benchmark of egocentric video-language assistants: https://fpv-iplab.github.io/Ego-EXTRA/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_13238 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Ego-EXTRA: video-language Egocentric Dataset for EXpert-TRAinee assistance Ragusa, Francesco Mazzamuto, Michele Forte, Rosario D'Ambra, Irene Fort, James Engel, Jakob Furnari, Antonino Farinella, Giovanni Maria Computer Vision and Pattern Recognition We present Ego-EXTRA, a video-language Egocentric Dataset for EXpert-TRAinee assistance. Ego-EXTRA features 50 hours of unscripted egocentric videos of subjects performing procedural activities (the trainees) while guided by real-world experts who provide guidance and answer specific questions using natural language. Following a ``Wizard of OZ'' data collection paradigm, the expert enacts a wearable intelligent assistant, looking at the activities performed by the trainee exclusively from their egocentric point of view, answering questions when asked by the trainee, or proactively interacting with suggestions during the procedures. This unique data collection protocol enables Ego-EXTRA to capture a high-quality dialogue in which expert-level feedback is provided to the trainee. Two-way dialogues between experts and trainees are recorded, transcribed, and used to create a novel benchmark comprising more than 15k high-quality Visual Question Answer sets, which we use to evaluate Multimodal Large Language Models. The results show that Ego-EXTRA is challenging and highlight the limitations of current models when used to provide expert-level assistance to the user. The Ego-EXTRA dataset is publicly available to support the benchmark of egocentric video-language assistants: https://fpv-iplab.github.io/Ego-EXTRA/. |
| title | Ego-EXTRA: video-language Egocentric Dataset for EXpert-TRAinee assistance |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2512.13238 |