Talk2Event: Grounded Understanding of Dynamic Scenes from Event Cameras
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914133080801280 |
|---|---|
| author | Kong, Lingdong Lu, Dongyue Liang, Ao Li, Rong Dong, Yuhao Hu, Tianshuai Ng, Lai Xing Ooi, Wei Tsang Cottereau, Benoit R. |
| author_facet | Kong, Lingdong Lu, Dongyue Liang, Ao Li, Rong Dong, Yuhao Hu, Tianshuai Ng, Lai Xing Ooi, Wei Tsang Cottereau, Benoit R. |
| contents | Event cameras offer microsecond-level latency and robustness to motion blur, making them ideal for understanding dynamic environments. Yet, connecting these asynchronous streams to human language remains an open challenge. We introduce Talk2Event, the first large-scale benchmark for language-driven object grounding in event-based perception. Built from real-world driving data, we provide over 30,000 validated referring expressions, each enriched with four grounding attributes -- appearance, status, relation to viewer, and relation to other objects -- bridging spatial, temporal, and relational reasoning. To fully exploit these cues, we propose EventRefer, an attribute-aware grounding framework that dynamically fuses multi-attribute representations through a Mixture of Event-Attribute Experts (MoEE). Our method adapts to different modalities and scene dynamics, achieving consistent gains over state-of-the-art baselines in event-only, frame-only, and event-frame fusion settings. We hope our dataset and approach will establish a foundation for advancing multimodal, temporally-aware, and language-driven perception in real-world robotics and autonomy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_17664 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Talk2Event: Grounded Understanding of Dynamic Scenes from Event Cameras Kong, Lingdong Lu, Dongyue Liang, Ao Li, Rong Dong, Yuhao Hu, Tianshuai Ng, Lai Xing Ooi, Wei Tsang Cottereau, Benoit R. Computer Vision and Pattern Recognition Robotics Event cameras offer microsecond-level latency and robustness to motion blur, making them ideal for understanding dynamic environments. Yet, connecting these asynchronous streams to human language remains an open challenge. We introduce Talk2Event, the first large-scale benchmark for language-driven object grounding in event-based perception. Built from real-world driving data, we provide over 30,000 validated referring expressions, each enriched with four grounding attributes -- appearance, status, relation to viewer, and relation to other objects -- bridging spatial, temporal, and relational reasoning. To fully exploit these cues, we propose EventRefer, an attribute-aware grounding framework that dynamically fuses multi-attribute representations through a Mixture of Event-Attribute Experts (MoEE). Our method adapts to different modalities and scene dynamics, achieving consistent gains over state-of-the-art baselines in event-only, frame-only, and event-frame fusion settings. We hope our dataset and approach will establish a foundation for advancing multimodal, temporally-aware, and language-driven perception in real-world robotics and autonomy. |
| title | Talk2Event: Grounded Understanding of Dynamic Scenes from Event Cameras |
| topic | Computer Vision and Pattern Recognition Robotics |
| url | https://arxiv.org/abs/2507.17664 |