Refine-by-Align: Reference-Guided Artifacts Refinement through Semantic Alignment
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913592001953792 |
|---|---|
| author | Song, Yizhi He, Liu Zhang, Zhifei Kim, Soo Ye Zhang, He Xiong, Wei Lin, Zhe Price, Brian Cohen, Scott Zhang, Jianming Aliaga, Daniel |
| author_facet | Song, Yizhi He, Liu Zhang, Zhifei Kim, Soo Ye Zhang, He Xiong, Wei Lin, Zhe Price, Brian Cohen, Scott Zhang, Jianming Aliaga, Daniel |
| contents | Personalized image generation has emerged from the recent advancements in generative models. However, these generated personalized images often suffer from localized artifacts such as incorrect logos, reducing fidelity and fine-grained identity details of the generated results. Furthermore, there is little prior work tackling this problem. To help improve these identity details in the personalized image generation, we introduce a new task: reference-guided artifacts refinement. We present Refine-by-Align, a first-of-its-kind model that employs a diffusion-based framework to address this challenge. Our model consists of two stages: Alignment Stage and Refinement Stage, which share weights of a unified neural network model. Given a generated image, a masked artifact region, and a reference image, the alignment stage identifies and extracts the corresponding regional features in the reference, which are then used by the refinement stage to fix the artifacts. Our model-agnostic pipeline requires no test-time tuning or optimization. It automatically enhances image fidelity and reference identity in the generated image, generalizing well to existing models on various tasks including but not limited to customization, generative compositing, view synthesis, and virtual try-on. Extensive experiments and comparisons demonstrate that our pipeline greatly pushes the boundary of fine details in the image synthesis models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_00306 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Refine-by-Align: Reference-Guided Artifacts Refinement through Semantic Alignment Song, Yizhi He, Liu Zhang, Zhifei Kim, Soo Ye Zhang, He Xiong, Wei Lin, Zhe Price, Brian Cohen, Scott Zhang, Jianming Aliaga, Daniel Computer Vision and Pattern Recognition Personalized image generation has emerged from the recent advancements in generative models. However, these generated personalized images often suffer from localized artifacts such as incorrect logos, reducing fidelity and fine-grained identity details of the generated results. Furthermore, there is little prior work tackling this problem. To help improve these identity details in the personalized image generation, we introduce a new task: reference-guided artifacts refinement. We present Refine-by-Align, a first-of-its-kind model that employs a diffusion-based framework to address this challenge. Our model consists of two stages: Alignment Stage and Refinement Stage, which share weights of a unified neural network model. Given a generated image, a masked artifact region, and a reference image, the alignment stage identifies and extracts the corresponding regional features in the reference, which are then used by the refinement stage to fix the artifacts. Our model-agnostic pipeline requires no test-time tuning or optimization. It automatically enhances image fidelity and reference identity in the generated image, generalizing well to existing models on various tasks including but not limited to customization, generative compositing, view synthesis, and virtual try-on. Extensive experiments and comparisons demonstrate that our pipeline greatly pushes the boundary of fine details in the image synthesis models. |
| title | Refine-by-Align: Reference-Guided Artifacts Refinement through Semantic Alignment |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.00306 |