Streaming Detection of Queried Event Start
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909416034402304 |
|---|---|
| author | Eyzaguirre, Cristobal Tang, Eric Buch, Shyamal Gaidon, Adrien Wu, Jiajun Niebles, Juan Carlos |
| author_facet | Eyzaguirre, Cristobal Tang, Eric Buch, Shyamal Gaidon, Adrien Wu, Jiajun Niebles, Juan Carlos |
| contents | Robotics, autonomous driving, augmented reality, and many embodied computer vision applications must quickly react to user-defined events unfolding in real time. We address this setting by proposing a novel task for multimodal video understanding-Streaming Detection of Queried Event Start (SDQES). The goal of SDQES is to identify the beginning of a complex event as described by a natural language query, with high accuracy and low latency. We introduce a new benchmark based on the Ego4D dataset, as well as new task-specific metrics to study streaming multimodal detection of diverse events in an egocentric video setting. Inspired by parameter-efficient fine-tuning methods in NLP and for video tasks, we propose adapter-based baselines that enable image-to-video transfer learning, allowing for efficient online video modeling. We evaluate three vision-language backbones and three adapter architectures on both short-clip and untrimmed video settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_03567 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Streaming Detection of Queried Event Start Eyzaguirre, Cristobal Tang, Eric Buch, Shyamal Gaidon, Adrien Wu, Jiajun Niebles, Juan Carlos Computer Vision and Pattern Recognition Robotics, autonomous driving, augmented reality, and many embodied computer vision applications must quickly react to user-defined events unfolding in real time. We address this setting by proposing a novel task for multimodal video understanding-Streaming Detection of Queried Event Start (SDQES). The goal of SDQES is to identify the beginning of a complex event as described by a natural language query, with high accuracy and low latency. We introduce a new benchmark based on the Ego4D dataset, as well as new task-specific metrics to study streaming multimodal detection of diverse events in an egocentric video setting. Inspired by parameter-efficient fine-tuning methods in NLP and for video tasks, we propose adapter-based baselines that enable image-to-video transfer learning, allowing for efficient online video modeling. We evaluate three vision-language backbones and three adapter architectures on both short-clip and untrimmed video settings. |
| title | Streaming Detection of Queried Event Start |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.03567 |