Mimir: Improving Video Diffusion Models for Precise Text Understanding
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915047871086592 |
|---|---|
| author | Tan, Shuai Gong, Biao Feng, Yutong Zheng, Kecheng Zheng, Dandan Shi, Shuwei Shen, Yujun Chen, Jingdong Yang, Ming |
| author_facet | Tan, Shuai Gong, Biao Feng, Yutong Zheng, Kecheng Zheng, Dandan Shi, Shuwei Shen, Yujun Chen, Jingdong Yang, Ming |
| contents | Text serves as the key control signal in video generation due to its narrative nature. To render text descriptions into video clips, current video diffusion models borrow features from text encoders yet struggle with limited text comprehension. The recent success of large language models (LLMs) showcases the power of decoder-only transformers, which offers three clear benefits for text-to-video (T2V) generation, namely, precise text understanding resulting from the superior scalability, imagination beyond the input text enabled by next token prediction, and flexibility to prioritize user interests through instruction tuning. Nevertheless, the feature distribution gap emerging from the two different text modeling paradigms hinders the direct use of LLMs in established T2V models. This work addresses this challenge with Mimir, an end-to-end training framework featuring a carefully tailored token fuser to harmonize the outputs from text encoders and LLMs. Such a design allows the T2V model to fully leverage learned video priors while capitalizing on the text-related capability of LLMs. Extensive quantitative and qualitative results demonstrate the effectiveness of Mimir in generating high-quality videos with excellent text comprehension, especially when processing short captions and managing shifting motions. Project page: https://lucaria-academy.github.io/Mimir/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_03085 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Mimir: Improving Video Diffusion Models for Precise Text Understanding Tan, Shuai Gong, Biao Feng, Yutong Zheng, Kecheng Zheng, Dandan Shi, Shuwei Shen, Yujun Chen, Jingdong Yang, Ming Computer Vision and Pattern Recognition Text serves as the key control signal in video generation due to its narrative nature. To render text descriptions into video clips, current video diffusion models borrow features from text encoders yet struggle with limited text comprehension. The recent success of large language models (LLMs) showcases the power of decoder-only transformers, which offers three clear benefits for text-to-video (T2V) generation, namely, precise text understanding resulting from the superior scalability, imagination beyond the input text enabled by next token prediction, and flexibility to prioritize user interests through instruction tuning. Nevertheless, the feature distribution gap emerging from the two different text modeling paradigms hinders the direct use of LLMs in established T2V models. This work addresses this challenge with Mimir, an end-to-end training framework featuring a carefully tailored token fuser to harmonize the outputs from text encoders and LLMs. Such a design allows the T2V model to fully leverage learned video priors while capitalizing on the text-related capability of LLMs. Extensive quantitative and qualitative results demonstrate the effectiveness of Mimir in generating high-quality videos with excellent text comprehension, especially when processing short captions and managing shifting motions. Project page: https://lucaria-academy.github.io/Mimir/ |
| title | Mimir: Improving Video Diffusion Models for Precise Text Understanding |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.03085 |