Towards Precise Scaling Laws for Video Diffusion Transformers
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909445391384576 |
|---|---|
| author | Yin, Yuanyang Zhao, Yaqi Zheng, Mingwu Lin, Ke Ou, Jiarong Chen, Rui Huang, Victor Shea-Jay Wang, Jiahao Tao, Xin Wan, Pengfei Zhang, Di Yin, Baoqun Zhang, Wentao Gai, Kun |
| author_facet | Yin, Yuanyang Zhao, Yaqi Zheng, Mingwu Lin, Ke Ou, Jiarong Chen, Rui Huang, Victor Shea-Jay Wang, Jiahao Tao, Xin Wan, Pengfei Zhang, Di Yin, Baoqun Zhang, Wentao Gai, Kun |
| contents | Achieving optimal performance of video diffusion transformers within given data and compute budget is crucial due to their high training costs. This necessitates precisely determining the optimal model size and training hyperparameters before large-scale training. While scaling laws are employed in language models to predict performance, their existence and accurate derivation in visual generation models remain underexplored. In this paper, we systematically analyze scaling laws for video diffusion transformers and confirm their presence. Moreover, we discover that, unlike language models, video diffusion models are more sensitive to learning rate and batch size, two hyperparameters often not precisely modeled. To address this, we propose a new scaling law that predicts optimal hyperparameters for any model size and compute budget. Under these optimal settings, we achieve comparable performance and reduce inference costs by 40.1% compared to conventional scaling methods, within a compute budget of 1e10 TFlops. Furthermore, we establish a more generalized and precise relationship among validation loss, any model size, and compute budget. This enables performance prediction for non-optimal model sizes, which may also be appealed under practical inference cost constraints, achieving a better trade-off. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_17470 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Precise Scaling Laws for Video Diffusion Transformers Yin, Yuanyang Zhao, Yaqi Zheng, Mingwu Lin, Ke Ou, Jiarong Chen, Rui Huang, Victor Shea-Jay Wang, Jiahao Tao, Xin Wan, Pengfei Zhang, Di Yin, Baoqun Zhang, Wentao Gai, Kun Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Achieving optimal performance of video diffusion transformers within given data and compute budget is crucial due to their high training costs. This necessitates precisely determining the optimal model size and training hyperparameters before large-scale training. While scaling laws are employed in language models to predict performance, their existence and accurate derivation in visual generation models remain underexplored. In this paper, we systematically analyze scaling laws for video diffusion transformers and confirm their presence. Moreover, we discover that, unlike language models, video diffusion models are more sensitive to learning rate and batch size, two hyperparameters often not precisely modeled. To address this, we propose a new scaling law that predicts optimal hyperparameters for any model size and compute budget. Under these optimal settings, we achieve comparable performance and reduce inference costs by 40.1% compared to conventional scaling methods, within a compute budget of 1e10 TFlops. Furthermore, we establish a more generalized and precise relationship among validation loss, any model size, and compute budget. This enables performance prediction for non-optimal model sizes, which may also be appealed under practical inference cost constraints, achieving a better trade-off. |
| title | Towards Precise Scaling Laws for Video Diffusion Transformers |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2411.17470 |