One-Minute Video Generation with Test-Time Training
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908305504337920 |
|---|---|
| author | Dalal, Karan Koceja, Daniel Hussein, Gashon Xu, Jiarui Zhao, Yue Song, Youjin Han, Shihao Cheung, Ka Chun Kautz, Jan Guestrin, Carlos Hashimoto, Tatsunori Koyejo, Sanmi Choi, Yejin Sun, Yu Wang, Xiaolong |
| author_facet | Dalal, Karan Koceja, Daniel Hussein, Gashon Xu, Jiarui Zhao, Yue Song, Youjin Han, Shihao Cheung, Ka Chun Kautz, Jan Guestrin, Carlos Hashimoto, Tatsunori Koyejo, Sanmi Choi, Yejin Sun, Yu Wang, Xiaolong |
| contents | Transformers today still struggle to generate one-minute videos because self-attention layers are inefficient for long context. Alternatives such as Mamba layers struggle with complex multi-scene stories because their hidden states are less expressive. We experiment with Test-Time Training (TTT) layers, whose hidden states themselves can be neural networks, therefore more expressive. Adding TTT layers into a pre-trained Transformer enables it to generate one-minute videos from text storyboards. For proof of concept, we curate a dataset based on Tom and Jerry cartoons. Compared to baselines such as Mamba~2, Gated DeltaNet, and sliding-window attention layers, TTT layers generate much more coherent videos that tell complex stories, leading by 34 Elo points in a human evaluation of 100 videos per method. Although promising, results still contain artifacts, likely due to the limited capability of the pre-trained 5B model. The efficiency of our implementation can also be improved. We have only experimented with one-minute videos due to resource constraints, but the approach can be extended to longer videos and more complex stories. Sample videos, code and annotations are available at: https://test-time-training.github.io/video-dit |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_05298 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | One-Minute Video Generation with Test-Time Training Dalal, Karan Koceja, Daniel Hussein, Gashon Xu, Jiarui Zhao, Yue Song, Youjin Han, Shihao Cheung, Ka Chun Kautz, Jan Guestrin, Carlos Hashimoto, Tatsunori Koyejo, Sanmi Choi, Yejin Sun, Yu Wang, Xiaolong Computer Vision and Pattern Recognition Transformers today still struggle to generate one-minute videos because self-attention layers are inefficient for long context. Alternatives such as Mamba layers struggle with complex multi-scene stories because their hidden states are less expressive. We experiment with Test-Time Training (TTT) layers, whose hidden states themselves can be neural networks, therefore more expressive. Adding TTT layers into a pre-trained Transformer enables it to generate one-minute videos from text storyboards. For proof of concept, we curate a dataset based on Tom and Jerry cartoons. Compared to baselines such as Mamba~2, Gated DeltaNet, and sliding-window attention layers, TTT layers generate much more coherent videos that tell complex stories, leading by 34 Elo points in a human evaluation of 100 videos per method. Although promising, results still contain artifacts, likely due to the limited capability of the pre-trained 5B model. The efficiency of our implementation can also be improved. We have only experimented with one-minute videos due to resource constraints, but the approach can be extended to longer videos and more complex stories. Sample videos, code and annotations are available at: https://test-time-training.github.io/video-dit |
| title | One-Minute Video Generation with Test-Time Training |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2504.05298 |