MoCha: Towards Movie-Grade Talking Character Synthesis
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910899095207936 |
|---|---|
| author | Wei, Cong Sun, Bo Ma, Haoyu Hou, Ji Juefei-Xu, Felix He, Zecheng Dai, Xiaoliang Zhang, Luxin Li, Kunpeng Hou, Tingbo Sinha, Animesh Vajda, Peter Chen, Wenhu |
| author_facet | Wei, Cong Sun, Bo Ma, Haoyu Hou, Ji Juefei-Xu, Felix He, Zecheng Dai, Xiaoliang Zhang, Luxin Li, Kunpeng Hou, Tingbo Sinha, Animesh Vajda, Peter Chen, Wenhu |
| contents | Recent advancements in video generation have achieved impressive motion realism, yet they often overlook character-driven storytelling, a crucial task for automated film, animation generation. We introduce Talking Characters, a more realistic task to generate talking character animations directly from speech and text. Unlike talking head, Talking Characters aims at generating the full portrait of one or more characters beyond the facial region. In this paper, we propose MoCha, the first of its kind to generate talking characters. To ensure precise synchronization between video and speech, we propose a speech-video window attention mechanism that effectively aligns speech and video tokens. To address the scarcity of large-scale speech-labeled video datasets, we introduce a joint training strategy that leverages both speech-labeled and text-labeled video data, significantly improving generalization across diverse character actions. We also design structured prompt templates with character tags, enabling, for the first time, multi-character conversation with turn-based dialogue-allowing AI-generated characters to engage in context-aware conversations with cinematic coherence. Extensive qualitative and quantitative evaluations, including human preference studies and benchmark comparisons, demonstrate that MoCha sets a new standard for AI-generated cinematic storytelling, achieving superior realism, expressiveness, controllability and generalization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_23307 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MoCha: Towards Movie-Grade Talking Character Synthesis Wei, Cong Sun, Bo Ma, Haoyu Hou, Ji Juefei-Xu, Felix He, Zecheng Dai, Xiaoliang Zhang, Luxin Li, Kunpeng Hou, Tingbo Sinha, Animesh Vajda, Peter Chen, Wenhu Computer Vision and Pattern Recognition Recent advancements in video generation have achieved impressive motion realism, yet they often overlook character-driven storytelling, a crucial task for automated film, animation generation. We introduce Talking Characters, a more realistic task to generate talking character animations directly from speech and text. Unlike talking head, Talking Characters aims at generating the full portrait of one or more characters beyond the facial region. In this paper, we propose MoCha, the first of its kind to generate talking characters. To ensure precise synchronization between video and speech, we propose a speech-video window attention mechanism that effectively aligns speech and video tokens. To address the scarcity of large-scale speech-labeled video datasets, we introduce a joint training strategy that leverages both speech-labeled and text-labeled video data, significantly improving generalization across diverse character actions. We also design structured prompt templates with character tags, enabling, for the first time, multi-character conversation with turn-based dialogue-allowing AI-generated characters to engage in context-aware conversations with cinematic coherence. Extensive qualitative and quantitative evaluations, including human preference studies and benchmark comparisons, demonstrate that MoCha sets a new standard for AI-generated cinematic storytelling, achieving superior realism, expressiveness, controllability and generalization. |
| title | MoCha: Towards Movie-Grade Talking Character Synthesis |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2503.23307 |