EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in the Wild
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916624055926784 |
|---|---|
| author | Kim, Junhyeok Kim, Min Soo Chung, Jiwan Cho, Jungbin Kim, Jisoo Kim, Sungwoong Sim, Gyeongbo Yu, Youngjae |
| author_facet | Kim, Junhyeok Kim, Min Soo Chung, Jiwan Cho, Jungbin Kim, Jisoo Kim, Sungwoong Sim, Gyeongbo Yu, Youngjae |
| contents | Predicting when to initiate speech in real-world environments remains a fundamental challenge for conversational agents. We introduce EgoSpeak, a novel framework for real-time speech initiation prediction in egocentric streaming video. By modeling the conversation from the speaker's first-person viewpoint, EgoSpeak is tailored for human-like interactions in which a conversational agent must continuously observe its environment and dynamically decide when to talk. Our approach bridges the gap between simplified experimental setups and complex natural conversations by integrating four key capabilities: (1) first-person perspective, (2) RGB processing, (3) online processing, and (4) untrimmed video processing. We also present YT-Conversation, a diverse collection of in-the-wild conversational videos from YouTube, as a resource for large-scale pretraining. Experiments on EasyCom and Ego4D demonstrate that EgoSpeak outperforms random and silence-based baselines in real time. Our results also highlight the importance of multimodal input and context length in effectively deciding when to speak. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_14892 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in the Wild Kim, Junhyeok Kim, Min Soo Chung, Jiwan Cho, Jungbin Kim, Jisoo Kim, Sungwoong Sim, Gyeongbo Yu, Youngjae Computer Vision and Pattern Recognition Artificial Intelligence Predicting when to initiate speech in real-world environments remains a fundamental challenge for conversational agents. We introduce EgoSpeak, a novel framework for real-time speech initiation prediction in egocentric streaming video. By modeling the conversation from the speaker's first-person viewpoint, EgoSpeak is tailored for human-like interactions in which a conversational agent must continuously observe its environment and dynamically decide when to talk. Our approach bridges the gap between simplified experimental setups and complex natural conversations by integrating four key capabilities: (1) first-person perspective, (2) RGB processing, (3) online processing, and (4) untrimmed video processing. We also present YT-Conversation, a diverse collection of in-the-wild conversational videos from YouTube, as a resource for large-scale pretraining. Experiments on EasyCom and Ego4D demonstrate that EgoSpeak outperforms random and silence-based baselines in real time. Our results also highlight the importance of multimodal input and context length in effectively deciding when to speak. |
| title | EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in the Wild |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2502.14892 |