Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.05364 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909284319625216 |
|---|---|
| author | Yun, Heeseung Gao, Ruohan Ananthabhotla, Ishwarya Kumar, Anurag Donley, Jacob Li, Chao Kim, Gunhee Ithapu, Vamsi Krishna Murdock, Calvin |
| author_facet | Yun, Heeseung Gao, Ruohan Ananthabhotla, Ishwarya Kumar, Anurag Donley, Jacob Li, Chao Kim, Gunhee Ithapu, Vamsi Krishna Murdock, Calvin |
| contents | Egocentric videos provide comprehensive contexts for user and scene understanding, spanning multisensory perception to behavioral interaction. We propose Spherical World-Locking (SWL) as a general framework for egocentric scene representation, which implicitly transforms multisensory streams with respect to measurements of head orientation. Compared to conventional head-locked egocentric representations with a 2D planar field-of-view, SWL effectively offsets challenges posed by self-motion, allowing for improved spatial synchronization between input modalities. Using a set of multisensory embeddings on a worldlocked sphere, we design a unified encoder-decoder transformer architecture that preserves the spherical structure of the scene representation, without requiring expensive projections between image and world coordinate systems. We evaluate the effectiveness of the proposed framework on multiple benchmark tasks for egocentric video understanding, including audio-visual active speaker localization, auditory spherical source localization, and behavior anticipation in everyday activities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_05364 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Spherical World-Locking for Audio-Visual Localization in Egocentric Videos Yun, Heeseung Gao, Ruohan Ananthabhotla, Ishwarya Kumar, Anurag Donley, Jacob Li, Chao Kim, Gunhee Ithapu, Vamsi Krishna Murdock, Calvin Computer Vision and Pattern Recognition Egocentric videos provide comprehensive contexts for user and scene understanding, spanning multisensory perception to behavioral interaction. We propose Spherical World-Locking (SWL) as a general framework for egocentric scene representation, which implicitly transforms multisensory streams with respect to measurements of head orientation. Compared to conventional head-locked egocentric representations with a 2D planar field-of-view, SWL effectively offsets challenges posed by self-motion, allowing for improved spatial synchronization between input modalities. Using a set of multisensory embeddings on a worldlocked sphere, we design a unified encoder-decoder transformer architecture that preserves the spherical structure of the scene representation, without requiring expensive projections between image and world coordinate systems. We evaluate the effectiveness of the proposed framework on multiple benchmark tasks for egocentric video understanding, including audio-visual active speaker localization, auditory spherical source localization, and behavior anticipation in everyday activities. |
| title | Spherical World-Locking for Audio-Visual Localization in Egocentric Videos |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.05364 |