Check, Locate, Rectify: A Training-Free Layout Calibration System for Text-to-Image Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916174086799360 |
|---|---|
| author | Gong, Biao Huang, Siteng Feng, Yutong Zhang, Shiwei Li, Yuyuan Liu, Yu |
| author_facet | Gong, Biao Huang, Siteng Feng, Yutong Zhang, Shiwei Li, Yuyuan Liu, Yu |
| contents | Diffusion models have recently achieved remarkable progress in generating realistic images. However, challenges remain in accurately understanding and synthesizing the layout requirements in the textual prompts. To align the generated image with layout instructions, we present a training-free layout calibration system SimM that intervenes in the generative process on the fly during inference time. Specifically, following a "check-locate-rectify" pipeline, the system first analyses the prompt to generate the target layout and compares it with the intermediate outputs to automatically detect errors. Then, by moving the located activations and making intra- and inter-map adjustments, the rectification process can be performed with negligible computational overhead. To evaluate SimM over a range of layout requirements, we present a benchmark SimMBench that compensates for the lack of superlative spatial relations in existing datasets. And both quantitative and qualitative results demonstrate the effectiveness of the proposed SimM in calibrating the layout inconsistencies. Our project page is at https://simm-t2i.github.io/SimM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_15773 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Check, Locate, Rectify: A Training-Free Layout Calibration System for Text-to-Image Generation Gong, Biao Huang, Siteng Feng, Yutong Zhang, Shiwei Li, Yuyuan Liu, Yu Computer Vision and Pattern Recognition Diffusion models have recently achieved remarkable progress in generating realistic images. However, challenges remain in accurately understanding and synthesizing the layout requirements in the textual prompts. To align the generated image with layout instructions, we present a training-free layout calibration system SimM that intervenes in the generative process on the fly during inference time. Specifically, following a "check-locate-rectify" pipeline, the system first analyses the prompt to generate the target layout and compares it with the intermediate outputs to automatically detect errors. Then, by moving the located activations and making intra- and inter-map adjustments, the rectification process can be performed with negligible computational overhead. To evaluate SimM over a range of layout requirements, we present a benchmark SimMBench that compensates for the lack of superlative spatial relations in existing datasets. And both quantitative and qualitative results demonstrate the effectiveness of the proposed SimM in calibrating the layout inconsistencies. Our project page is at https://simm-t2i.github.io/SimM. |
| title | Check, Locate, Rectify: A Training-Free Layout Calibration System for Text-to-Image Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2311.15773 |