When ControlNet Meets Inexplicit Masks: A Case Study of ControlNet on its Contour-following Ability
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912072860696576 |
|---|---|
| author | Xuan, Wenjie Xu, Yufei Zhao, Shanshan Wang, Chaoyue Liu, Juhua Du, Bo Tao, Dacheng |
| author_facet | Xuan, Wenjie Xu, Yufei Zhao, Shanshan Wang, Chaoyue Liu, Juhua Du, Bo Tao, Dacheng |
| contents | ControlNet excels at creating content that closely matches precise contours in user-provided masks. However, when these masks contain noise, as a frequent occurrence with non-expert users, the output would include unwanted artifacts. This paper first highlights the crucial role of controlling the impact of these inexplicit masks with diverse deterioration levels through in-depth analysis. Subsequently, to enhance controllability with inexplicit masks, an advanced Shape-aware ControlNet consisting of a deterioration estimator and a shape-prior modulation block is devised. The deterioration estimator assesses the deterioration factor of the provided masks. Then this factor is utilized in the modulation block to adaptively modulate the model's contour-following ability, which helps it dismiss the noise part in the inexplicit masks. Extensive experiments prove its effectiveness in encouraging ControlNet to interpret inaccurate spatial conditions robustly rather than blindly following the given contours, suitable for diverse kinds of conditions. We showcase application scenarios like modifying shape priors and composable shape-controllable generation. Codes are available at github. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_00467 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | When ControlNet Meets Inexplicit Masks: A Case Study of ControlNet on its Contour-following Ability Xuan, Wenjie Xu, Yufei Zhao, Shanshan Wang, Chaoyue Liu, Juhua Du, Bo Tao, Dacheng Computer Vision and Pattern Recognition ControlNet excels at creating content that closely matches precise contours in user-provided masks. However, when these masks contain noise, as a frequent occurrence with non-expert users, the output would include unwanted artifacts. This paper first highlights the crucial role of controlling the impact of these inexplicit masks with diverse deterioration levels through in-depth analysis. Subsequently, to enhance controllability with inexplicit masks, an advanced Shape-aware ControlNet consisting of a deterioration estimator and a shape-prior modulation block is devised. The deterioration estimator assesses the deterioration factor of the provided masks. Then this factor is utilized in the modulation block to adaptively modulate the model's contour-following ability, which helps it dismiss the noise part in the inexplicit masks. Extensive experiments prove its effectiveness in encouraging ControlNet to interpret inaccurate spatial conditions robustly rather than blindly following the given contours, suitable for diverse kinds of conditions. We showcase application scenarios like modifying shape priors and composable shape-controllable generation. Codes are available at github. |
| title | When ControlNet Meets Inexplicit Masks: A Case Study of ControlNet on its Contour-following Ability |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.00467 |