Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.13248 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929526108323840 |
|---|---|
| author | Yuan, Zhengqing Liu, Yixin Cao, Yihan Sun, Weixiang Jia, Haolong Chen, Ruoxi Li, Zhaoxu Lin, Bin Yuan, Li He, Lifang Wang, Chi Ye, Yanfang Sun, Lichao |
| author_facet | Yuan, Zhengqing Liu, Yixin Cao, Yihan Sun, Weixiang Jia, Haolong Chen, Ruoxi Li, Zhaoxu Lin, Bin Yuan, Li He, Lifang Wang, Chi Ye, Yanfang Sun, Lichao |
| contents | Text-to-video generation has made significant strides, but replicating the capabilities of advanced systems like OpenAI Sora remains challenging due to their closed-source nature. Existing open-source methods struggle to achieve comparable performance, often hindered by ineffective agent collaboration and inadequate training data quality. In this paper, we introduce Mora, a novel multi-agent framework that leverages existing open-source modules to replicate Sora functionalities. We address these fundamental limitations by proposing three key techniques: (1) multi-agent fine-tuning with a self-modulation factor to enhance inter-agent coordination, (2) a data-free training strategy that uses large models to synthesize training data, and (3) a human-in-the-loop mechanism combined with multimodal large language models for data filtering to ensure high-quality training datasets. Our comprehensive experiments on six video generation tasks demonstrate that Mora achieves performance comparable to Sora on VBench, outperforming existing open-source methods across various tasks. Specifically, in the text-to-video generation task, Mora achieved a Video Quality score of 0.800, surpassing Sora 0.797 and outperforming all other baseline models across six key metrics. Additionally, in the image-to-video generation task, Mora achieved a perfect Dynamic Degree score of 1.00, demonstrating exceptional capability in enhancing motion realism and achieving higher Imaging Quality than Sora. These results highlight the potential of collaborative multi-agent systems and human-in-the-loop mechanisms in advancing text-to-video generation. Our code is available at \url{https://github.com/lichao-sun/Mora}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_13248 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Mora: Enabling Generalist Video Generation via A Multi-Agent Framework Yuan, Zhengqing Liu, Yixin Cao, Yihan Sun, Weixiang Jia, Haolong Chen, Ruoxi Li, Zhaoxu Lin, Bin Yuan, Li He, Lifang Wang, Chi Ye, Yanfang Sun, Lichao Computer Vision and Pattern Recognition Text-to-video generation has made significant strides, but replicating the capabilities of advanced systems like OpenAI Sora remains challenging due to their closed-source nature. Existing open-source methods struggle to achieve comparable performance, often hindered by ineffective agent collaboration and inadequate training data quality. In this paper, we introduce Mora, a novel multi-agent framework that leverages existing open-source modules to replicate Sora functionalities. We address these fundamental limitations by proposing three key techniques: (1) multi-agent fine-tuning with a self-modulation factor to enhance inter-agent coordination, (2) a data-free training strategy that uses large models to synthesize training data, and (3) a human-in-the-loop mechanism combined with multimodal large language models for data filtering to ensure high-quality training datasets. Our comprehensive experiments on six video generation tasks demonstrate that Mora achieves performance comparable to Sora on VBench, outperforming existing open-source methods across various tasks. Specifically, in the text-to-video generation task, Mora achieved a Video Quality score of 0.800, surpassing Sora 0.797 and outperforming all other baseline models across six key metrics. Additionally, in the image-to-video generation task, Mora achieved a perfect Dynamic Degree score of 1.00, demonstrating exceptional capability in enhancing motion realism and achieving higher Imaging Quality than Sora. These results highlight the potential of collaborative multi-agent systems and human-in-the-loop mechanisms in advancing text-to-video generation. Our code is available at \url{https://github.com/lichao-sun/Mora}. |
| title | Mora: Enabling Generalist Video Generation via A Multi-Agent Framework |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.13248 |