AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918489343655936 |
|---|---|
| author | Zheng, Kai Kang, Zejian Mao, Rui Zou, Hongyuan Fei, Yuanchen Xu, Xuanyang Huang, Xiangru |
| author_facet | Zheng, Kai Kang, Zejian Mao, Rui Zou, Hongyuan Fei, Yuanchen Xu, Xuanyang Huang, Xiangru |
| contents | Speech-driven facial animation requires accurate correspondence between acoustic signals and facial motion, especially for articulation-related mouth movements. However, directly mapping speech audio to facial coefficients often overlooks the linguistic and phonetic structure underlying speech production. In this paper, we propose AudioFace, a language-assisted framework for speech-driven blendshape generation that treats mouth-related facial coefficient prediction as a structured generation problem guided by linguistic and articulatory information. Instead of relying solely on acoustic features, our method leverages the prior knowledge of multimodal large language models and introduces transcript- and phoneme-level cues to bridge speech signals with interpretable facial actions. Extensive experiments show that AudioFace achieves superior performance across multiple evaluation metrics, validating the effectiveness of language-assisted and multimodal-prior-guided speech-driven facial animation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_07478 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models Zheng, Kai Kang, Zejian Mao, Rui Zou, Hongyuan Fei, Yuanchen Xu, Xuanyang Huang, Xiangru Computer Vision and Pattern Recognition Speech-driven facial animation requires accurate correspondence between acoustic signals and facial motion, especially for articulation-related mouth movements. However, directly mapping speech audio to facial coefficients often overlooks the linguistic and phonetic structure underlying speech production. In this paper, we propose AudioFace, a language-assisted framework for speech-driven blendshape generation that treats mouth-related facial coefficient prediction as a structured generation problem guided by linguistic and articulatory information. Instead of relying solely on acoustic features, our method leverages the prior knowledge of multimodal large language models and introduces transcript- and phoneme-level cues to bridge speech signals with interpretable facial actions. Extensive experiments show that AudioFace achieves superior performance across multiple evaluation metrics, validating the effectiveness of language-assisted and multimodal-prior-guided speech-driven facial animation. |
| title | AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2605.07478 |