Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.13352 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913630507761664 |
|---|---|
| author | An, Jingkun Zhu, Yinghao Li, Zongjian Zhou, Enshen Feng, Haoran Huang, Xijie Chen, Bohua Shi, Yemin Pan, Chengwei |
| author_facet | An, Jingkun Zhu, Yinghao Li, Zongjian Zhou, Enshen Feng, Haoran Huang, Xijie Chen, Bohua Shi, Yemin Pan, Chengwei |
| contents | Text-to-Image (T2I) diffusion models have achieved remarkable success in image generation. Despite their progress, challenges remain in both prompt-following ability, image quality and lack of high-quality datasets, which are essential for refining these models. As acquiring labeled data is costly, we introduce AGFSync, a framework that enhances T2I diffusion models through Direct Preference Optimization (DPO) in a fully AI-driven approach. AGFSync utilizes Vision-Language Models (VLM) to assess image quality across style, coherence, and aesthetics, generating feedback data within an AI-driven loop. By applying AGFSync to leading T2I models such as SD v1.4, v1.5, and SDXL-base, our extensive experiments on the TIFA dataset demonstrate notable improvements in VQA scores, aesthetic evaluations, and performance on the HPSv2 benchmark, consistently outperforming the base models. AGFSync's method of refining T2I diffusion models paves the way for scalable alignment techniques. Our code and dataset are publicly available at https://anjingkun.github.io/AGFSync. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_13352 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AGFSync: Leveraging AI-Generated Feedback for Preference Optimization in Text-to-Image Generation An, Jingkun Zhu, Yinghao Li, Zongjian Zhou, Enshen Feng, Haoran Huang, Xijie Chen, Bohua Shi, Yemin Pan, Chengwei Computer Vision and Pattern Recognition Text-to-Image (T2I) diffusion models have achieved remarkable success in image generation. Despite their progress, challenges remain in both prompt-following ability, image quality and lack of high-quality datasets, which are essential for refining these models. As acquiring labeled data is costly, we introduce AGFSync, a framework that enhances T2I diffusion models through Direct Preference Optimization (DPO) in a fully AI-driven approach. AGFSync utilizes Vision-Language Models (VLM) to assess image quality across style, coherence, and aesthetics, generating feedback data within an AI-driven loop. By applying AGFSync to leading T2I models such as SD v1.4, v1.5, and SDXL-base, our extensive experiments on the TIFA dataset demonstrate notable improvements in VQA scores, aesthetic evaluations, and performance on the HPSv2 benchmark, consistently outperforming the base models. AGFSync's method of refining T2I diffusion models paves the way for scalable alignment techniques. Our code and dataset are publicly available at https://anjingkun.github.io/AGFSync. |
| title | AGFSync: Leveraging AI-Generated Feedback for Preference Optimization in Text-to-Image Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.13352 |