Proactive Assistant Dialogue Generation from Streaming Egocentric Videos
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913881479184384 |
|---|---|
| author | Zhang, Yichi Dong, Xin Luna Lin, Zhaojiang Madotto, Andrea Kumar, Anuj Damavandi, Babak Chai, Joyce Moon, Seungwhan |
| author_facet | Zhang, Yichi Dong, Xin Luna Lin, Zhaojiang Madotto, Andrea Kumar, Anuj Damavandi, Babak Chai, Joyce Moon, Seungwhan |
| contents | Recent advances in conversational AI have been substantial, but developing real-time systems for perceptual task guidance remains challenging. These systems must provide interactive, proactive assistance based on streaming visual inputs, yet their development is constrained by the costly and labor-intensive process of data collection and system evaluation. To address these limitations, we present a comprehensive framework with three key contributions. First, we introduce a novel data curation pipeline that synthesizes dialogues from annotated egocentric videos, resulting in \dataset, a large-scale synthetic dialogue dataset spanning multiple domains. Second, we develop a suite of automatic evaluation metrics, validated through extensive human studies. Third, we propose an end-to-end model that processes streaming video inputs to generate contextually appropriate responses, incorporating novel techniques for handling data imbalance and long-duration videos. This work lays the foundation for developing real-time, proactive AI assistants capable of guiding users through diverse tasks. Project page: https://pro-assist.github.io/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_05904 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Proactive Assistant Dialogue Generation from Streaming Egocentric Videos Zhang, Yichi Dong, Xin Luna Lin, Zhaojiang Madotto, Andrea Kumar, Anuj Damavandi, Babak Chai, Joyce Moon, Seungwhan Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Human-Computer Interaction Recent advances in conversational AI have been substantial, but developing real-time systems for perceptual task guidance remains challenging. These systems must provide interactive, proactive assistance based on streaming visual inputs, yet their development is constrained by the costly and labor-intensive process of data collection and system evaluation. To address these limitations, we present a comprehensive framework with three key contributions. First, we introduce a novel data curation pipeline that synthesizes dialogues from annotated egocentric videos, resulting in \dataset, a large-scale synthetic dialogue dataset spanning multiple domains. Second, we develop a suite of automatic evaluation metrics, validated through extensive human studies. Third, we propose an end-to-end model that processes streaming video inputs to generate contextually appropriate responses, incorporating novel techniques for handling data imbalance and long-duration videos. This work lays the foundation for developing real-time, proactive AI assistants capable of guiding users through diverse tasks. Project page: https://pro-assist.github.io/ |
| title | Proactive Assistant Dialogue Generation from Streaming Egocentric Videos |
| topic | Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Human-Computer Interaction |
| url | https://arxiv.org/abs/2506.05904 |