Inf-DiT: Upsampling Any-Resolution Image with Memory-Efficient Diffusion Transformer
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911870469799936 |
|---|---|
| author | Yang, Zhuoyi Jiang, Heyang Hong, Wenyi Teng, Jiayan Zheng, Wendi Dong, Yuxiao Ding, Ming Tang, Jie |
| author_facet | Yang, Zhuoyi Jiang, Heyang Hong, Wenyi Teng, Jiayan Zheng, Wendi Dong, Yuxiao Ding, Ming Tang, Jie |
| contents | Diffusion models have shown remarkable performance in image generation in recent years. However, due to a quadratic increase in memory during generating ultra-high-resolution images (e.g. 4096*4096), the resolution of generated images is often limited to 1024*1024. In this work. we propose a unidirectional block attention mechanism that can adaptively adjust the memory overhead during the inference process and handle global dependencies. Building on this module, we adopt the DiT structure for upsampling and develop an infinite super-resolution model capable of upsampling images of various shapes and resolutions. Comprehensive experiments show that our model achieves SOTA performance in generating ultra-high-resolution images in both machine and human evaluation. Compared to commonly used UNet structures, our model can save more than 5x memory when generating 4096*4096 images. The project URL is https://github.com/THUDM/Inf-DiT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_04312 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Inf-DiT: Upsampling Any-Resolution Image with Memory-Efficient Diffusion Transformer Yang, Zhuoyi Jiang, Heyang Hong, Wenyi Teng, Jiayan Zheng, Wendi Dong, Yuxiao Ding, Ming Tang, Jie Computer Vision and Pattern Recognition Diffusion models have shown remarkable performance in image generation in recent years. However, due to a quadratic increase in memory during generating ultra-high-resolution images (e.g. 4096*4096), the resolution of generated images is often limited to 1024*1024. In this work. we propose a unidirectional block attention mechanism that can adaptively adjust the memory overhead during the inference process and handle global dependencies. Building on this module, we adopt the DiT structure for upsampling and develop an infinite super-resolution model capable of upsampling images of various shapes and resolutions. Comprehensive experiments show that our model achieves SOTA performance in generating ultra-high-resolution images in both machine and human evaluation. Compared to commonly used UNet structures, our model can save more than 5x memory when generating 4096*4096 images. The project URL is https://github.com/THUDM/Inf-DiT. |
| title | Inf-DiT: Upsampling Any-Resolution Image with Memory-Efficient Diffusion Transformer |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2405.04312 |