LEDiT: Your Length-Extrapolatable Diffusion Transformer without Positional Encoding
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916966058426368 |
|---|---|
| author | Zhang, Shen Liang, Siyuan Tan, Yaning Chen, Zhaowei Li, Linze Wu, Ge Chen, Yuhao Li, Shuheng Zhao, Zhenyu Chen, Caihua Liang, Jiajun Tang, Yao |
| author_facet | Zhang, Shen Liang, Siyuan Tan, Yaning Chen, Zhaowei Li, Linze Wu, Ge Chen, Yuhao Li, Shuheng Zhao, Zhenyu Chen, Caihua Liang, Jiajun Tang, Yao |
| contents | Diffusion transformers (DiTs) struggle to generate images at resolutions higher than their training resolutions. The primary obstacle is that the explicit positional encodings(PE), such as RoPE, need extrapolating to unseen positions which degrades performance when the inference resolution differs from training. In this paper, We propose a Length-Extrapolatable Diffusion Transformer~(LEDiT) to overcome this limitation. LEDiT needs no explicit PEs, thereby avoiding PE extrapolation. The key innovation of LEDiT lies in the use of causal attention. We demonstrate that causal attention can implicitly encode global positional information and show that such information facilitates extrapolation. We further introduce a locality enhancement module, which captures fine-grained local information to complement the global coarse-grained position information encoded by causal attention. Experimental results on both conditional and text-to-image generation tasks demonstrate that LEDiT supports up to 4x resolution scaling (e.g., from 256x256 to 512x512), achieving better image quality compared to the state-of-the-art length extrapolation methods. We believe that LEDiT marks a departure from the standard RoPE-based methods and offers a promising insight into length extrapolation. Project page: https://shenzhang2145.github.io/ledit/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_04344 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LEDiT: Your Length-Extrapolatable Diffusion Transformer without Positional Encoding Zhang, Shen Liang, Siyuan Tan, Yaning Chen, Zhaowei Li, Linze Wu, Ge Chen, Yuhao Li, Shuheng Zhao, Zhenyu Chen, Caihua Liang, Jiajun Tang, Yao Computer Vision and Pattern Recognition Diffusion transformers (DiTs) struggle to generate images at resolutions higher than their training resolutions. The primary obstacle is that the explicit positional encodings(PE), such as RoPE, need extrapolating to unseen positions which degrades performance when the inference resolution differs from training. In this paper, We propose a Length-Extrapolatable Diffusion Transformer~(LEDiT) to overcome this limitation. LEDiT needs no explicit PEs, thereby avoiding PE extrapolation. The key innovation of LEDiT lies in the use of causal attention. We demonstrate that causal attention can implicitly encode global positional information and show that such information facilitates extrapolation. We further introduce a locality enhancement module, which captures fine-grained local information to complement the global coarse-grained position information encoded by causal attention. Experimental results on both conditional and text-to-image generation tasks demonstrate that LEDiT supports up to 4x resolution scaling (e.g., from 256x256 to 512x512), achieving better image quality compared to the state-of-the-art length extrapolation methods. We believe that LEDiT marks a departure from the standard RoPE-based methods and offers a promising insight into length extrapolation. Project page: https://shenzhang2145.github.io/ledit/ |
| title | LEDiT: Your Length-Extrapolatable Diffusion Transformer without Positional Encoding |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2503.04344 |