VividVoice: A Unified Framework for Scene-Aware Visually-Driven Speech Synthesis
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910009653198848 |
|---|---|
| author | Ma, Chengyuan Jin, Jiawei Xiong, Ruijie Jin, Chunxiang Yan, Canxiang Yang, Wenming |
| author_facet | Ma, Chengyuan Jin, Jiawei Xiong, Ruijie Jin, Chunxiang Yan, Canxiang Yang, Wenming |
| contents | We introduce and define a novel task-Scene-Aware Visually-Driven Speech Synthesis, aimed at addressing the limitations of existing speech generation models in creating immersive auditory experiences that align with the real physical world. To tackle the two core challenges of data scarcity and modality decoupling, we propose VividVoice, a unified generative framework. First, we constructed a large-scale, high-quality hybrid multimodal dataset, Vivid-210K, which, through an innovative programmatic pipeline, establishes a strong correlation between visual scenes, speaker identity, and audio for the first time. Second, we designed a core alignment module, D-MSVA, which leverages a decoupled memory bank architecture and a cross-modal hybrid supervision strategy to achieve fine-grained alignment from visual scenes to timbre and environmental acoustic features. Both subjective and objective experimental results provide strong evidence that VividVoice significantly outperforms existing baseline models in terms of audio fidelity, content clarity, and multimodal consistency. Our demo is available at https://chengyuann.github.io/VividVoice/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_02591 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | VividVoice: A Unified Framework for Scene-Aware Visually-Driven Speech Synthesis Ma, Chengyuan Jin, Jiawei Xiong, Ruijie Jin, Chunxiang Yan, Canxiang Yang, Wenming Sound Artificial Intelligence We introduce and define a novel task-Scene-Aware Visually-Driven Speech Synthesis, aimed at addressing the limitations of existing speech generation models in creating immersive auditory experiences that align with the real physical world. To tackle the two core challenges of data scarcity and modality decoupling, we propose VividVoice, a unified generative framework. First, we constructed a large-scale, high-quality hybrid multimodal dataset, Vivid-210K, which, through an innovative programmatic pipeline, establishes a strong correlation between visual scenes, speaker identity, and audio for the first time. Second, we designed a core alignment module, D-MSVA, which leverages a decoupled memory bank architecture and a cross-modal hybrid supervision strategy to achieve fine-grained alignment from visual scenes to timbre and environmental acoustic features. Both subjective and objective experimental results provide strong evidence that VividVoice significantly outperforms existing baseline models in terms of audio fidelity, content clarity, and multimodal consistency. Our demo is available at https://chengyuann.github.io/VividVoice/. |
| title | VividVoice: A Unified Framework for Scene-Aware Visually-Driven Speech Synthesis |
| topic | Sound Artificial Intelligence |
| url | https://arxiv.org/abs/2602.02591 |