SpongeBob: Sync-Aware Harmonious Audio-Visual Generative Editing
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910272330924032 |
|---|---|
| author | Liang, Sen Wang, Cong Guan, Fengbin Yu, Zhentao Lu, Yiting Wang, Yuanzhi Zhou, Yuan Li, Xin Chen, Zhibo |
| author_facet | Liang, Sen Wang, Cong Guan, Fengbin Yu, Zhentao Lu, Yiting Wang, Yuanzhi Zhou, Yuan Li, Xin Chen, Zhibo |
| contents | Visual and acoustic events in the physical world are inherently coupled, yet existing video editing methods typically adopt decoupled pipelines, lacking bidirectional modality interaction. This results in two key limitations: (i) audio-visual desynchronization and (ii) contextual conflicts between generated audio and preserved content. To address these, we propose SpongeBob, the first end-to-end audio-visual joint editing framework featuring bidirectional cross-modal interaction. For synchronization, a Sync-Aware Mechanism aligns visual edits with sound events via bidirectional attention, temporal alignment, and spatial constraints. For contextual consistency, a Context-Aware Module leverages acoustic and visual context attention to prevent semantic clashes. Additionally, we introduce Sync-Preserving Training and Guidance (SPTG) to enhance alignment without degrading quality. Due to the scarcity of paired data, we construct a scalable data pipeline and a large-scale subject-level dataset. We also propose SpongeBob-Bench for systematic evaluation. Experiments show SpongeBob significantly outperforms existing baselines, improving Sync-C by 30% and Ctx-F1 by 12.5%. Our project page is available at: https://hy-spongebob.github.io/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_25193 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SpongeBob: Sync-Aware Harmonious Audio-Visual Generative Editing Liang, Sen Wang, Cong Guan, Fengbin Yu, Zhentao Lu, Yiting Wang, Yuanzhi Zhou, Yuan Li, Xin Chen, Zhibo Computer Vision and Pattern Recognition Visual and acoustic events in the physical world are inherently coupled, yet existing video editing methods typically adopt decoupled pipelines, lacking bidirectional modality interaction. This results in two key limitations: (i) audio-visual desynchronization and (ii) contextual conflicts between generated audio and preserved content. To address these, we propose SpongeBob, the first end-to-end audio-visual joint editing framework featuring bidirectional cross-modal interaction. For synchronization, a Sync-Aware Mechanism aligns visual edits with sound events via bidirectional attention, temporal alignment, and spatial constraints. For contextual consistency, a Context-Aware Module leverages acoustic and visual context attention to prevent semantic clashes. Additionally, we introduce Sync-Preserving Training and Guidance (SPTG) to enhance alignment without degrading quality. Due to the scarcity of paired data, we construct a scalable data pipeline and a large-scale subject-level dataset. We also propose SpongeBob-Bench for systematic evaluation. Experiments show SpongeBob significantly outperforms existing baselines, improving Sync-C by 30% and Ctx-F1 by 12.5%. Our project page is available at: https://hy-spongebob.github.io/. |
| title | SpongeBob: Sync-Aware Harmonious Audio-Visual Generative Editing |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2605.25193 |