ByTheWay: Boost Your Text-to-Video Generation Model to Higher Quality in a Training-free Way
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910846971543552 |
|---|---|
| author | Bu, Jiazi Ling, Pengyang Zhang, Pan Wu, Tong Dong, Xiaoyi Zang, Yuhang Cao, Yuhang Lin, Dahua Wang, Jiaqi |
| author_facet | Bu, Jiazi Ling, Pengyang Zhang, Pan Wu, Tong Dong, Xiaoyi Zang, Yuhang Cao, Yuhang Lin, Dahua Wang, Jiaqi |
| contents | The text-to-video (T2V) generation models, offering convenient visual creation, have recently garnered increasing attention. Despite their substantial potential, the generated videos may present artifacts, including structural implausibility, temporal inconsistency, and a lack of motion, often resulting in near-static video. In this work, we have identified a correlation between the disparity of temporal attention maps across different blocks and the occurrence of temporal inconsistencies. Additionally, we have observed that the energy contained within the temporal attention maps is directly related to the magnitude of motion amplitude in the generated videos. Based on these observations, we present ByTheWay, a training-free method to improve the quality of text-to-video generation without introducing additional parameters, augmenting memory or sampling time. Specifically, ByTheWay is composed of two principal components: 1) Temporal Self-Guidance improves the structural plausibility and temporal consistency of generated videos by reducing the disparity between the temporal attention maps across various decoder blocks. 2) Fourier-based Motion Enhancement enhances the magnitude and richness of motion by amplifying the energy of the map. Extensive experiments demonstrate that ByTheWay significantly improves the quality of text-to-video generation with negligible additional cost. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_06241 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ByTheWay: Boost Your Text-to-Video Generation Model to Higher Quality in a Training-free Way Bu, Jiazi Ling, Pengyang Zhang, Pan Wu, Tong Dong, Xiaoyi Zang, Yuhang Cao, Yuhang Lin, Dahua Wang, Jiaqi Computer Vision and Pattern Recognition The text-to-video (T2V) generation models, offering convenient visual creation, have recently garnered increasing attention. Despite their substantial potential, the generated videos may present artifacts, including structural implausibility, temporal inconsistency, and a lack of motion, often resulting in near-static video. In this work, we have identified a correlation between the disparity of temporal attention maps across different blocks and the occurrence of temporal inconsistencies. Additionally, we have observed that the energy contained within the temporal attention maps is directly related to the magnitude of motion amplitude in the generated videos. Based on these observations, we present ByTheWay, a training-free method to improve the quality of text-to-video generation without introducing additional parameters, augmenting memory or sampling time. Specifically, ByTheWay is composed of two principal components: 1) Temporal Self-Guidance improves the structural plausibility and temporal consistency of generated videos by reducing the disparity between the temporal attention maps across various decoder blocks. 2) Fourier-based Motion Enhancement enhances the magnitude and richness of motion by amplifying the energy of the map. Extensive experiments demonstrate that ByTheWay significantly improves the quality of text-to-video generation with negligible additional cost. |
| title | ByTheWay: Boost Your Text-to-Video Generation Model to Higher Quality in a Training-free Way |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.06241 |