Layout-Conditioned Autoregressive Text-to-Image Generation via Structured Masking
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911155616743424 |
|---|---|
| author | Zheng, Zirui Isobe, Takashi Shen, Tong Jia, Xu Zhao, Jianbin Li, Xiaomin Ge, Mengmeng Li, Baolu Wang, Qinghe Li, Dong Zhou, Dong Zhuge, Yunzhi Lu, Huchuan Barsoum, Emad |
| author_facet | Zheng, Zirui Isobe, Takashi Shen, Tong Jia, Xu Zhao, Jianbin Li, Xiaomin Ge, Mengmeng Li, Baolu Wang, Qinghe Li, Dong Zhou, Dong Zhuge, Yunzhi Lu, Huchuan Barsoum, Emad |
| contents | While autoregressive (AR) models have demonstrated remarkable success in image generation, extending them to layout-conditioned generation remains challenging due to the sparse nature of layout conditions and the risk of feature entanglement. We present Structured Masking for AR-based Layout-to-Image (SMARLI), a novel framework for layoutto-image generation that effectively integrates spatial layout constraints into AR-based image generation. To equip AR model with layout control, a specially designed structured masking strategy is applied to attention computation to govern the interaction among the global prompt, layout, and image tokens. This design prevents mis-association between different regions and their descriptions while enabling sufficient injection of layout constraints into the generation process. To further enhance generation quality and layout accuracy, we incorporate Group Relative Policy Optimization (GRPO) based post-training scheme with specially designed layout reward functions for next-set-based AR models. Experimental results demonstrate that SMARLI is able to seamlessly integrate layout tokens with text and image tokens without compromising generation quality. It achieves superior layoutaware control while maintaining the structural simplicity and generation efficiency of AR models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_12046 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Layout-Conditioned Autoregressive Text-to-Image Generation via Structured Masking Zheng, Zirui Isobe, Takashi Shen, Tong Jia, Xu Zhao, Jianbin Li, Xiaomin Ge, Mengmeng Li, Baolu Wang, Qinghe Li, Dong Zhou, Dong Zhuge, Yunzhi Lu, Huchuan Barsoum, Emad Computer Vision and Pattern Recognition Artificial Intelligence While autoregressive (AR) models have demonstrated remarkable success in image generation, extending them to layout-conditioned generation remains challenging due to the sparse nature of layout conditions and the risk of feature entanglement. We present Structured Masking for AR-based Layout-to-Image (SMARLI), a novel framework for layoutto-image generation that effectively integrates spatial layout constraints into AR-based image generation. To equip AR model with layout control, a specially designed structured masking strategy is applied to attention computation to govern the interaction among the global prompt, layout, and image tokens. This design prevents mis-association between different regions and their descriptions while enabling sufficient injection of layout constraints into the generation process. To further enhance generation quality and layout accuracy, we incorporate Group Relative Policy Optimization (GRPO) based post-training scheme with specially designed layout reward functions for next-set-based AR models. Experimental results demonstrate that SMARLI is able to seamlessly integrate layout tokens with text and image tokens without compromising generation quality. It achieves superior layoutaware control while maintaining the structural simplicity and generation efficiency of AR models. |
| title | Layout-Conditioned Autoregressive Text-to-Image Generation via Structured Masking |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2509.12046 |