Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.17630 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909439218417664 |
|---|---|
| author | Xu, Jiamin Zheng, Yuxin Li, Zelong Wang, Chi Gu, Renshu Xu, Weiwei Xu, Gang |
| author_facet | Xu, Jiamin Zheng, Yuxin Li, Zelong Wang, Chi Gu, Renshu Xu, Weiwei Xu, Gang |
| contents | Achieving high-quality shadow removal with strong generalizability is challenging in scenes with complex global illumination. Due to the limited diversity in shadow removal datasets, current methods are prone to overfitting training data, often leading to reduced performance on unseen cases. To address this, we leverage the rich visual priors of a pre-trained Stable Diffusion (SD) model and propose a two-stage fine-tuning pipeline to adapt the SD model for stable and efficient shadow removal. In the first stage, we fix the VAE and fine-tune the denoiser in latent space, which yields substantial shadow removal but may lose some high-frequency details. To resolve this, we introduce a second stage, called the detail injection stage. This stage selectively extracts features from the VAE encoder to modulate the decoder, injecting fine details into the final results. Experimental results show that our method outperforms state-of-the-art shadow removal techniques. The cross-dataset evaluation further demonstrates that our method generalizes effectively to unseen data, enhancing the applicability of shadow removal methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_17630 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Detail-Preserving Latent Diffusion for Stable Shadow Removal Xu, Jiamin Zheng, Yuxin Li, Zelong Wang, Chi Gu, Renshu Xu, Weiwei Xu, Gang Computer Vision and Pattern Recognition Achieving high-quality shadow removal with strong generalizability is challenging in scenes with complex global illumination. Due to the limited diversity in shadow removal datasets, current methods are prone to overfitting training data, often leading to reduced performance on unseen cases. To address this, we leverage the rich visual priors of a pre-trained Stable Diffusion (SD) model and propose a two-stage fine-tuning pipeline to adapt the SD model for stable and efficient shadow removal. In the first stage, we fix the VAE and fine-tune the denoiser in latent space, which yields substantial shadow removal but may lose some high-frequency details. To resolve this, we introduce a second stage, called the detail injection stage. This stage selectively extracts features from the VAE encoder to modulate the decoder, injecting fine details into the final results. Experimental results show that our method outperforms state-of-the-art shadow removal techniques. The cross-dataset evaluation further demonstrates that our method generalizes effectively to unseen data, enhancing the applicability of shadow removal methods. |
| title | Detail-Preserving Latent Diffusion for Stable Shadow Removal |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.17630 |