DiffDoctor: Diagnosing Image Diffusion Models Before Treating
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929752243175424 |
|---|---|
| author | Wang, Yiyang Chen, Xi Xu, Xiaogang Ji, Sihui Liu, Yu Shen, Yujun Zhao, Hengshuang |
| author_facet | Wang, Yiyang Chen, Xi Xu, Xiaogang Ji, Sihui Liu, Yu Shen, Yujun Zhao, Hengshuang |
| contents | In spite of recent progress, image diffusion models still produce artifacts. A common solution is to leverage the feedback provided by quality assessment systems or human annotators to optimize the model, where images are generally rated in their entirety. In this work, we believe problem-solving starts with identification, yielding the request that the model should be aware of not just the presence of defects in an image, but their specific locations. Motivated by this, we propose DiffDoctor, a two-stage pipeline to assist image diffusion models in generating fewer artifacts. Concretely, the first stage targets developing a robust artifact detector, for which we collect a dataset of over 1M flawed synthesized images and set up an efficient human-in-the-loop annotation process, incorporating a carefully designed class-balance strategy. The learned artifact detector is then involved in the second stage to optimize the diffusion model by providing pixel-level feedback. Extensive experiments on text-to-image diffusion models demonstrate the effectiveness of our artifact detector as well as the soundness of our diagnose-then-treat design. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_12382 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DiffDoctor: Diagnosing Image Diffusion Models Before Treating Wang, Yiyang Chen, Xi Xu, Xiaogang Ji, Sihui Liu, Yu Shen, Yujun Zhao, Hengshuang Computer Vision and Pattern Recognition In spite of recent progress, image diffusion models still produce artifacts. A common solution is to leverage the feedback provided by quality assessment systems or human annotators to optimize the model, where images are generally rated in their entirety. In this work, we believe problem-solving starts with identification, yielding the request that the model should be aware of not just the presence of defects in an image, but their specific locations. Motivated by this, we propose DiffDoctor, a two-stage pipeline to assist image diffusion models in generating fewer artifacts. Concretely, the first stage targets developing a robust artifact detector, for which we collect a dataset of over 1M flawed synthesized images and set up an efficient human-in-the-loop annotation process, incorporating a carefully designed class-balance strategy. The learned artifact detector is then involved in the second stage to optimize the diffusion model by providing pixel-level feedback. Extensive experiments on text-to-image diffusion models demonstrate the effectiveness of our artifact detector as well as the soundness of our diagnose-then-treat design. |
| title | DiffDoctor: Diagnosing Image Diffusion Models Before Treating |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2501.12382 |