Online Episodic Memory Visual Query Localization with Egocentric Streaming Object Memory
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917158540279808 |
|---|---|
| author | Manigrasso, Zaira Dunnhofer, Matteo Furnari, Antonino Nottebaum, Moritz Finocchiaro, Antonio Marana, Davide Forte, Rosario Farinella, Giovanni Maria Micheloni, Christian |
| author_facet | Manigrasso, Zaira Dunnhofer, Matteo Furnari, Antonino Nottebaum, Moritz Finocchiaro, Antonio Marana, Davide Forte, Rosario Farinella, Giovanni Maria Micheloni, Christian |
| contents | Episodic memory retrieval enables wearable cameras to recall objects or events previously observed in video. However, existing formulations assume an "offline" setting with full video access at query time, limiting their applicability in real-world scenarios with power and storage-constrained wearable devices. Towards more application-ready episodic memory systems, we introduce Online Visual Query 2D (OVQ2D), a task where models process video streams online, observing each frame only once, and retrieve object localizations using a compact memory instead of full video history. We address OVQ2D with ESOM (Egocentric Streaming Object Memory), a novel framework integrating an object discovery module, an object tracking module, and a memory module that find, track, and store spatio-temporal object information for efficient querying. Experiments on Ego4D demonstrate ESOM's superiority over other online approaches, though OVQ2D remains challenging, with top performance at only ~4% success. ESOM's accuracy increases markedly with perfect object tracking (31.91%), discovery (40.55%), or both (81.92%), underscoring the need of applied research on these components. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_16934 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Online Episodic Memory Visual Query Localization with Egocentric Streaming Object Memory Manigrasso, Zaira Dunnhofer, Matteo Furnari, Antonino Nottebaum, Moritz Finocchiaro, Antonio Marana, Davide Forte, Rosario Farinella, Giovanni Maria Micheloni, Christian Computer Vision and Pattern Recognition Episodic memory retrieval enables wearable cameras to recall objects or events previously observed in video. However, existing formulations assume an "offline" setting with full video access at query time, limiting their applicability in real-world scenarios with power and storage-constrained wearable devices. Towards more application-ready episodic memory systems, we introduce Online Visual Query 2D (OVQ2D), a task where models process video streams online, observing each frame only once, and retrieve object localizations using a compact memory instead of full video history. We address OVQ2D with ESOM (Egocentric Streaming Object Memory), a novel framework integrating an object discovery module, an object tracking module, and a memory module that find, track, and store spatio-temporal object information for efficient querying. Experiments on Ego4D demonstrate ESOM's superiority over other online approaches, though OVQ2D remains challenging, with top performance at only ~4% success. ESOM's accuracy increases markedly with perfect object tracking (31.91%), discovery (40.55%), or both (81.92%), underscoring the need of applied research on these components. |
| title | Online Episodic Memory Visual Query Localization with Egocentric Streaming Object Memory |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2411.16934 |