Dual-Stream Decoupled Learning for Temporal Consistency and Speaker Interaction in AVSD
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911596986499072 |
|---|---|
| author | Xiao, Junhao Feng, Shun Wu, Zhiyu Yu, Jinghan Yao, Haibiao Ma, Zhiyuan Li, Jianjun Bao, Youjun Chen, Yi |
| author_facet | Xiao, Junhao Feng, Shun Wu, Zhiyu Yu, Jinghan Yao, Haibiao Ma, Zhiyuan Li, Jianjun Bao, Youjun Chen, Yi |
| contents | Audio-Visual Speaker Detection (AVSD) hinges on modeling both individual temporal continuity and inter-personal social context. Existing coupled architectures struggle to reconcile these tasks in shared representation spaces due to conflicting inductive biases: temporal modeling favors low-frequency smoothness, while inter-personal interaction requires high-frequency discriminability. We propose D$^2$Stream, a decoupled dual-stream framework that explicitly isolates these functionalities into parallel, task-specific branches. Specifically, the Intra-speaker Temporal Continuity (ITC) stream captures longitudinal stability, whereas the Inter-personal Social Relation (ISR) stream models transversal social cues. Quantitative gradient analysis reveals an evolutionary divergence in update directions, stabilizing at 86.1°, which confirms the inherent task conflict and the effectiveness of our structural decoupling. D$^2$Stream breaks the long-standing performance plateau, achieving a state-of-the-art 95.6% mAP on AVA-ActiveSpeaker and superior generalization on Columbia ASD, all within a lightweight and efficient design. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_19130 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Dual-Stream Decoupled Learning for Temporal Consistency and Speaker Interaction in AVSD Xiao, Junhao Feng, Shun Wu, Zhiyu Yu, Jinghan Yao, Haibiao Ma, Zhiyuan Li, Jianjun Bao, Youjun Chen, Yi Multimedia Audio-Visual Speaker Detection (AVSD) hinges on modeling both individual temporal continuity and inter-personal social context. Existing coupled architectures struggle to reconcile these tasks in shared representation spaces due to conflicting inductive biases: temporal modeling favors low-frequency smoothness, while inter-personal interaction requires high-frequency discriminability. We propose D$^2$Stream, a decoupled dual-stream framework that explicitly isolates these functionalities into parallel, task-specific branches. Specifically, the Intra-speaker Temporal Continuity (ITC) stream captures longitudinal stability, whereas the Inter-personal Social Relation (ISR) stream models transversal social cues. Quantitative gradient analysis reveals an evolutionary divergence in update directions, stabilizing at 86.1°, which confirms the inherent task conflict and the effectiveness of our structural decoupling. D$^2$Stream breaks the long-standing performance plateau, achieving a state-of-the-art 95.6% mAP on AVA-ActiveSpeaker and superior generalization on Columbia ASD, all within a lightweight and efficient design. |
| title | Dual-Stream Decoupled Learning for Temporal Consistency and Speaker Interaction in AVSD |
| topic | Multimedia |
| url | https://arxiv.org/abs/2512.19130 |