xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909302015393792 |
|---|---|
| author | Qin, Can Xia, Congying Ramakrishnan, Krithika Ryoo, Michael Tu, Lifu Feng, Yihao Shu, Manli Zhou, Honglu Awadalla, Anas Wang, Jun Purushwalkam, Senthil Xue, Le Zhou, Yingbo Wang, Huan Savarese, Silvio Niebles, Juan Carlos Chen, Zeyuan Xu, Ran Xiong, Caiming |
| author_facet | Qin, Can Xia, Congying Ramakrishnan, Krithika Ryoo, Michael Tu, Lifu Feng, Yihao Shu, Manli Zhou, Honglu Awadalla, Anas Wang, Jun Purushwalkam, Senthil Xue, Le Zhou, Yingbo Wang, Huan Savarese, Silvio Niebles, Juan Carlos Chen, Zeyuan Xu, Ran Xiong, Caiming |
| contents | We present xGen-VideoSyn-1, a text-to-video (T2V) generation model capable of producing realistic scenes from textual descriptions. Building on recent advancements, such as OpenAI's Sora, we explore the latent diffusion model (LDM) architecture and introduce a video variational autoencoder (VidVAE). VidVAE compresses video data both spatially and temporally, significantly reducing the length of visual tokens and the computational demands associated with generating long-sequence videos. To further address the computational costs, we propose a divide-and-merge strategy that maintains temporal consistency across video segments. Our Diffusion Transformer (DiT) model incorporates spatial and temporal self-attention layers, enabling robust generalization across different timeframes and aspect ratios. We have devised a data processing pipeline from the very beginning and collected over 13M high-quality video-text pairs. The pipeline includes multiple steps such as clipping, text detection, motion estimation, aesthetics scoring, and dense captioning based on our in-house video-LLM model. Training the VidVAE and DiT models required approximately 40 and 642 H100 days, respectively. Our model supports over 14-second 720p video generation in an end-to-end way and demonstrates competitive performance against state-of-the-art T2V models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_12590 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations Qin, Can Xia, Congying Ramakrishnan, Krithika Ryoo, Michael Tu, Lifu Feng, Yihao Shu, Manli Zhou, Honglu Awadalla, Anas Wang, Jun Purushwalkam, Senthil Xue, Le Zhou, Yingbo Wang, Huan Savarese, Silvio Niebles, Juan Carlos Chen, Zeyuan Xu, Ran Xiong, Caiming Computer Vision and Pattern Recognition Artificial Intelligence We present xGen-VideoSyn-1, a text-to-video (T2V) generation model capable of producing realistic scenes from textual descriptions. Building on recent advancements, such as OpenAI's Sora, we explore the latent diffusion model (LDM) architecture and introduce a video variational autoencoder (VidVAE). VidVAE compresses video data both spatially and temporally, significantly reducing the length of visual tokens and the computational demands associated with generating long-sequence videos. To further address the computational costs, we propose a divide-and-merge strategy that maintains temporal consistency across video segments. Our Diffusion Transformer (DiT) model incorporates spatial and temporal self-attention layers, enabling robust generalization across different timeframes and aspect ratios. We have devised a data processing pipeline from the very beginning and collected over 13M high-quality video-text pairs. The pipeline includes multiple steps such as clipping, text detection, motion estimation, aesthetics scoring, and dense captioning based on our in-house video-LLM model. Training the VidVAE and DiT models required approximately 40 and 642 H100 days, respectively. Our model supports over 14-second 720p video generation in an end-to-end way and demonstrates competitive performance against state-of-the-art T2V models. |
| title | xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2408.12590 |