HiCAST: Highly Customized Arbitrary Style Transfer with Adapter Enhanced Diffusion Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929207094804480 |
|---|---|
| author | Wang, Hanzhang Wang, Haoran Yang, Jinze Yu, Zhongrui Xie, Zeke Tian, Lei Xiao, Xinyan Jiang, Junjun Liu, Xianming Sun, Mingming |
| author_facet | Wang, Hanzhang Wang, Haoran Yang, Jinze Yu, Zhongrui Xie, Zeke Tian, Lei Xiao, Xinyan Jiang, Junjun Liu, Xianming Sun, Mingming |
| contents | The goal of Arbitrary Style Transfer (AST) is injecting the artistic features of a style reference into a given image/video. Existing methods usually focus on pursuing the balance between style and content, whereas ignoring the significant demand for flexible and customized stylization results and thereby limiting their practical application. To address this critical issue, a novel AST approach namely HiCAST is proposed, which is capable of explicitly customizing the stylization results according to various source of semantic clues. In the specific, our model is constructed based on Latent Diffusion Model (LDM) and elaborately designed to absorb content and style instance as conditions of LDM. It is characterized by introducing of \textit{Style Adapter}, which allows user to flexibly manipulate the output results by aligning multi-level style information and intrinsic knowledge in LDM. Lastly, we further extend our model to perform video AST. A novel learning objective is leveraged for video diffusion model training, which significantly improve cross-frame temporal consistency in the premise of maintaining stylization strength. Qualitative and quantitative comparisons as well as comprehensive user studies demonstrate that our HiCAST outperforms the existing SoTA methods in generating visually plausible stylization results. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_05870 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | HiCAST: Highly Customized Arbitrary Style Transfer with Adapter Enhanced Diffusion Models Wang, Hanzhang Wang, Haoran Yang, Jinze Yu, Zhongrui Xie, Zeke Tian, Lei Xiao, Xinyan Jiang, Junjun Liu, Xianming Sun, Mingming Computer Vision and Pattern Recognition Artificial Intelligence The goal of Arbitrary Style Transfer (AST) is injecting the artistic features of a style reference into a given image/video. Existing methods usually focus on pursuing the balance between style and content, whereas ignoring the significant demand for flexible and customized stylization results and thereby limiting their practical application. To address this critical issue, a novel AST approach namely HiCAST is proposed, which is capable of explicitly customizing the stylization results according to various source of semantic clues. In the specific, our model is constructed based on Latent Diffusion Model (LDM) and elaborately designed to absorb content and style instance as conditions of LDM. It is characterized by introducing of \textit{Style Adapter}, which allows user to flexibly manipulate the output results by aligning multi-level style information and intrinsic knowledge in LDM. Lastly, we further extend our model to perform video AST. A novel learning objective is leveraged for video diffusion model training, which significantly improve cross-frame temporal consistency in the premise of maintaining stylization strength. Qualitative and quantitative comparisons as well as comprehensive user studies demonstrate that our HiCAST outperforms the existing SoTA methods in generating visually plausible stylization results. |
| title | HiCAST: Highly Customized Arbitrary Style Transfer with Adapter Enhanced Diffusion Models |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2401.05870 |