Multimodal Semantic-Aware Automatic Colorization with Diffusion Prior
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914770527977472 |
|---|---|
| author | Wang, Han Chai, Xinning Wang, Yiwen Zhang, Yuhong Xie, Rong Song, Li |
| author_facet | Wang, Han Chai, Xinning Wang, Yiwen Zhang, Yuhong Xie, Rong Song, Li |
| contents | Colorizing grayscale images offers an engaging visual experience. Existing automatic colorization methods often fail to generate satisfactory results due to incorrect semantic colors and unsaturated colors. In this work, we propose an automatic colorization pipeline to overcome these challenges. We leverage the extraordinary generative ability of the diffusion prior to synthesize color with plausible semantics. To overcome the artifacts introduced by the diffusion prior, we apply the luminance conditional guidance. Moreover, we adopt multimodal high-level semantic priors to help the model understand the image content and deliver saturated colors. Besides, a luminance-aware decoder is designed to restore details and enhance overall visual quality. The proposed pipeline synthesizes saturated colors while maintaining plausible semantics. Experiments indicate that our proposed method considers both diversity and fidelity, surpassing previous methods in terms of perceptual realism and gain most human preference. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_16678 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multimodal Semantic-Aware Automatic Colorization with Diffusion Prior Wang, Han Chai, Xinning Wang, Yiwen Zhang, Yuhong Xie, Rong Song, Li Computer Vision and Pattern Recognition Colorizing grayscale images offers an engaging visual experience. Existing automatic colorization methods often fail to generate satisfactory results due to incorrect semantic colors and unsaturated colors. In this work, we propose an automatic colorization pipeline to overcome these challenges. We leverage the extraordinary generative ability of the diffusion prior to synthesize color with plausible semantics. To overcome the artifacts introduced by the diffusion prior, we apply the luminance conditional guidance. Moreover, we adopt multimodal high-level semantic priors to help the model understand the image content and deliver saturated colors. Besides, a luminance-aware decoder is designed to restore details and enhance overall visual quality. The proposed pipeline synthesizes saturated colors while maintaining plausible semantics. Experiments indicate that our proposed method considers both diversity and fidelity, surpassing previous methods in terms of perceptual realism and gain most human preference. |
| title | Multimodal Semantic-Aware Automatic Colorization with Diffusion Prior |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2404.16678 |