Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.23132 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917359600533504 |
|---|---|
| author | Pan, Dongwei Guo, Longwei Guan, Jiazhi Huang, Luying Li, Yiding Liu, Haojie Feng, Haocheng He, Wei Wang, Kaisiyuan Zhou, Hang |
| author_facet | Pan, Dongwei Guo, Longwei Guan, Jiazhi Huang, Luying Li, Yiding Liu, Haojie Feng, Haocheng He, Wei Wang, Kaisiyuan Zhou, Hang |
| contents | Despite progress in speech-to-video synthesis, existing methods often struggle to capture cross-individual dependencies and provide fine-grained control over reactive behaviors in dyadic settings. To address these challenges, we propose InterDyad, a framework that enables naturalistic interactive dynamics synthesis via querying structural motion guidance. Specifically, we first design an Interactivity Injector that achieves video reenactment based on identity-agnostic motion priors extracted from reference videos. Building upon this, we introduce a MetaQuery-based modality alignment mechanism to bridge the gap between conversational audio and these motion priors. By leveraging a Multimodal Large Language Model (MLLM), our framework is able to distill linguistic intent from audio to dictate the precise timing and appropriateness of reactions. To further improve lip-sync quality under extreme head poses, we propose Role-aware Dyadic Gaussian Guidance (RoDG) for enhanced lip-synchronization and spatial consistency. Finally, we introduce a dedicated evaluation suite with novelly designed metrics to quantify dyadic interaction. Comprehensive experiments demonstrate that InterDyad significantly outperforms state-of-the-art methods in producing natural and contextually grounded two-person interactions. Please refer to our project page for demo videos: https://interdyad.github.io/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_23132 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | InterDyad: Interactive Dyadic Speech-to-Video Generation by Querying Intermediate Visual Guidance Pan, Dongwei Guo, Longwei Guan, Jiazhi Huang, Luying Li, Yiding Liu, Haojie Feng, Haocheng He, Wei Wang, Kaisiyuan Zhou, Hang Computer Vision and Pattern Recognition Despite progress in speech-to-video synthesis, existing methods often struggle to capture cross-individual dependencies and provide fine-grained control over reactive behaviors in dyadic settings. To address these challenges, we propose InterDyad, a framework that enables naturalistic interactive dynamics synthesis via querying structural motion guidance. Specifically, we first design an Interactivity Injector that achieves video reenactment based on identity-agnostic motion priors extracted from reference videos. Building upon this, we introduce a MetaQuery-based modality alignment mechanism to bridge the gap between conversational audio and these motion priors. By leveraging a Multimodal Large Language Model (MLLM), our framework is able to distill linguistic intent from audio to dictate the precise timing and appropriateness of reactions. To further improve lip-sync quality under extreme head poses, we propose Role-aware Dyadic Gaussian Guidance (RoDG) for enhanced lip-synchronization and spatial consistency. Finally, we introduce a dedicated evaluation suite with novelly designed metrics to quantify dyadic interaction. Comprehensive experiments demonstrate that InterDyad significantly outperforms state-of-the-art methods in producing natural and contextually grounded two-person interactions. Please refer to our project page for demo videos: https://interdyad.github.io/. |
| title | InterDyad: Interactive Dyadic Speech-to-Video Generation by Querying Intermediate Visual Guidance |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2603.23132 |