CueNet: Robust Audio-Visual Speaker Extraction through Cross-Modal Cue Mining and Interaction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917305572655104 |
|---|---|
| author | Wang, Jiadong Zhang, Ke Qian, Xinyuan Tao, Ruijie Li, Haizhou Schuller, Björn |
| author_facet | Wang, Jiadong Zhang, Ke Qian, Xinyuan Tao, Ruijie Li, Haizhou Schuller, Björn |
| contents | Audio-visual speaker extraction has attracted increasing attention, as it removes the need for pre-registered speech and leverages the visual modality as a complement to audio. Although existing methods have achieved impressive performance, the issue of degraded visual inputs has received relatively little attention, despite being common in real-world scenarios. Previous attempts to address this problem have mainly involved training with degraded visual data. However, visual degradation can occur in many unpredictable ways, making it impractical to simulate all possible cases during training. In this paper, we aim to enhance the robustness of audio-visual speaker extraction against impaired visual inputs without relying on degraded videos during training. Inspired by observations from human perceptual mechanisms, we propose an audio-visual learner that disentangles speaker information, acoustic synchronisation, and semantic synchronisation as distinct cues. Furthermore, we design a dedicated interaction module that effectively integrates these cues to provide a reliable guidance signal for speaker extraction. Extensive experiments demonstrate the strong robustness of the proposed model under various visual degradations and its clear superiority over existing methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_01530 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | CueNet: Robust Audio-Visual Speaker Extraction through Cross-Modal Cue Mining and Interaction Wang, Jiadong Zhang, Ke Qian, Xinyuan Tao, Ruijie Li, Haizhou Schuller, Björn Multimedia Audio-visual speaker extraction has attracted increasing attention, as it removes the need for pre-registered speech and leverages the visual modality as a complement to audio. Although existing methods have achieved impressive performance, the issue of degraded visual inputs has received relatively little attention, despite being common in real-world scenarios. Previous attempts to address this problem have mainly involved training with degraded visual data. However, visual degradation can occur in many unpredictable ways, making it impractical to simulate all possible cases during training. In this paper, we aim to enhance the robustness of audio-visual speaker extraction against impaired visual inputs without relying on degraded videos during training. Inspired by observations from human perceptual mechanisms, we propose an audio-visual learner that disentangles speaker information, acoustic synchronisation, and semantic synchronisation as distinct cues. Furthermore, we design a dedicated interaction module that effectively integrates these cues to provide a reliable guidance signal for speaker extraction. Extensive experiments demonstrate the strong robustness of the proposed model under various visual degradations and its clear superiority over existing methods. |
| title | CueNet: Robust Audio-Visual Speaker Extraction through Cross-Modal Cue Mining and Interaction |
| topic | Multimedia |
| url | https://arxiv.org/abs/2603.01530 |