SINGER: Vivid Audio-driven Singing Video Generation with Multi-scale Spectral Diffusion Model
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910727193755648 |
|---|---|
| author | Li, Yan Zhou, Ziya Wang, Zhiqiang Xue, Wei Luo, Wenhan Guo, Yike |
| author_facet | Li, Yan Zhou, Ziya Wang, Zhiqiang Xue, Wei Luo, Wenhan Guo, Yike |
| contents | Recent advancements in generative models have significantly enhanced talking face video generation, yet singing video generation remains underexplored. The differences between human talking and singing limit the performance of existing talking face video generation models when applied to singing. The fundamental differences between talking and singing-specifically in audio characteristics and behavioral expressions-limit the effectiveness of existing models. We observe that the differences between singing and talking audios manifest in terms of frequency and amplitude. To address this, we have designed a multi-scale spectral module to help the model learn singing patterns in the spectral domain. Additionally, we develop a spectral-filtering module that aids the model in learning the human behaviors associated with singing audio. These two modules are integrated into the diffusion model to enhance singing video generation performance, resulting in our proposed model, SINGER. Furthermore, the lack of high-quality real-world singing face videos has hindered the development of the singing video generation community. To address this gap, we have collected an in-the-wild audio-visual singing dataset to facilitate research in this area. Our experiments demonstrate that SINGER is capable of generating vivid singing videos and outperforms state-of-the-art methods in both objective and subjective evaluations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_03430 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SINGER: Vivid Audio-driven Singing Video Generation with Multi-scale Spectral Diffusion Model Li, Yan Zhou, Ziya Wang, Zhiqiang Xue, Wei Luo, Wenhan Guo, Yike Computer Vision and Pattern Recognition Machine Learning Sound Recent advancements in generative models have significantly enhanced talking face video generation, yet singing video generation remains underexplored. The differences between human talking and singing limit the performance of existing talking face video generation models when applied to singing. The fundamental differences between talking and singing-specifically in audio characteristics and behavioral expressions-limit the effectiveness of existing models. We observe that the differences between singing and talking audios manifest in terms of frequency and amplitude. To address this, we have designed a multi-scale spectral module to help the model learn singing patterns in the spectral domain. Additionally, we develop a spectral-filtering module that aids the model in learning the human behaviors associated with singing audio. These two modules are integrated into the diffusion model to enhance singing video generation performance, resulting in our proposed model, SINGER. Furthermore, the lack of high-quality real-world singing face videos has hindered the development of the singing video generation community. To address this gap, we have collected an in-the-wild audio-visual singing dataset to facilitate research in this area. Our experiments demonstrate that SINGER is capable of generating vivid singing videos and outperforms state-of-the-art methods in both objective and subjective evaluations. |
| title | SINGER: Vivid Audio-driven Singing Video Generation with Multi-scale Spectral Diffusion Model |
| topic | Computer Vision and Pattern Recognition Machine Learning Sound |
| url | https://arxiv.org/abs/2412.03430 |