Synchformer: Efficient Synchronization from Sparse Cues
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910310477070336 |
|---|---|
| author | Iashin, Vladimir Xie, Weidi Rahtu, Esa Zisserman, Andrew |
| author_facet | Iashin, Vladimir Xie, Weidi Rahtu, Esa Zisserman, Andrew |
| contents | Our objective is audio-visual synchronization with a focus on 'in-the-wild' videos, such as those on YouTube, where synchronization cues can be sparse. Our contributions include a novel audio-visual synchronization model, and training that decouples feature extraction from synchronization modelling through multi-modal segment-level contrastive pre-training. This approach achieves state-of-the-art performance in both dense and sparse settings. We also extend synchronization model training to AudioSet a million-scale 'in-the-wild' dataset, investigate evidence attribution techniques for interpretability, and explore a new capability for synchronization models: audio-visual synchronizability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_16423 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Synchformer: Efficient Synchronization from Sparse Cues Iashin, Vladimir Xie, Weidi Rahtu, Esa Zisserman, Andrew Computer Vision and Pattern Recognition Machine Learning Multimedia Sound Audio and Speech Processing Our objective is audio-visual synchronization with a focus on 'in-the-wild' videos, such as those on YouTube, where synchronization cues can be sparse. Our contributions include a novel audio-visual synchronization model, and training that decouples feature extraction from synchronization modelling through multi-modal segment-level contrastive pre-training. This approach achieves state-of-the-art performance in both dense and sparse settings. We also extend synchronization model training to AudioSet a million-scale 'in-the-wild' dataset, investigate evidence attribution techniques for interpretability, and explore a new capability for synchronization models: audio-visual synchronizability. |
| title | Synchformer: Efficient Synchronization from Sparse Cues |
| topic | Computer Vision and Pattern Recognition Machine Learning Multimedia Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2401.16423 |