Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.10127 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917971087065088 |
|---|---|
| author | He, Runze Cheng, Bo Ma, Yuhang Jia, Qingxiang Liu, Shanyuan Ma, Ao Wu, Xiaoyu Wu, Liebucha Leng, Dawei Yin, Yuhui |
| author_facet | He, Runze Cheng, Bo Ma, Yuhang Jia, Qingxiang Liu, Shanyuan Ma, Ao Wu, Xiaoyu Wu, Liebucha Leng, Dawei Yin, Yuhui |
| contents | In this paper, we propose a unified layout planning and image generation model, PlanGen, which can pre-plan spatial layout conditions before generating images. Unlike previous diffusion-based models that treat layout planning and layout-to-image as two separate models, PlanGen jointly models the two tasks into one autoregressive transformer using only next-token prediction. PlanGen integrates layout conditions into the model as context without requiring specialized encoding of local captions and bounding box coordinates, which provides significant advantages over the previous embed-and-pool operations on layout conditions, particularly when dealing with complex layouts. Unified prompting allows PlanGen to perform multitasking training related to layout, including layout planning, layout-to-image generation, image layout understanding, etc. In addition, PlanGen can be seamlessly expanded to layout-guided image manipulation thanks to the well-designed modeling, with teacher-forcing content manipulation policy and negative layout guidance. Extensive experiments verify the effectiveness of our PlanGen in multiple layoutrelated tasks, showing its great potential. Code is available at: https://360cvgroup.github.io/PlanGen. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_10127 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PlanGen: Towards Unified Layout Planning and Image Generation in Auto-Regressive Vision Language Models He, Runze Cheng, Bo Ma, Yuhang Jia, Qingxiang Liu, Shanyuan Ma, Ao Wu, Xiaoyu Wu, Liebucha Leng, Dawei Yin, Yuhui Computer Vision and Pattern Recognition In this paper, we propose a unified layout planning and image generation model, PlanGen, which can pre-plan spatial layout conditions before generating images. Unlike previous diffusion-based models that treat layout planning and layout-to-image as two separate models, PlanGen jointly models the two tasks into one autoregressive transformer using only next-token prediction. PlanGen integrates layout conditions into the model as context without requiring specialized encoding of local captions and bounding box coordinates, which provides significant advantages over the previous embed-and-pool operations on layout conditions, particularly when dealing with complex layouts. Unified prompting allows PlanGen to perform multitasking training related to layout, including layout planning, layout-to-image generation, image layout understanding, etc. In addition, PlanGen can be seamlessly expanded to layout-guided image manipulation thanks to the well-designed modeling, with teacher-forcing content manipulation policy and negative layout guidance. Extensive experiments verify the effectiveness of our PlanGen in multiple layoutrelated tasks, showing its great potential. Code is available at: https://360cvgroup.github.io/PlanGen. |
| title | PlanGen: Towards Unified Layout Planning and Image Generation in Auto-Regressive Vision Language Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2503.10127 |