OmniControlNet: Dual-stage Integration for Conditional Image Generation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910479192948736 |
|---|---|
| author | Wang, Yilin Xu, Haiyang Zhang, Xiang Chen, Zeyuan Sha, Zhizhou Wang, Zirui Tu, Zhuowen |
| author_facet | Wang, Yilin Xu, Haiyang Zhang, Xiang Chen, Zeyuan Sha, Zhizhou Wang, Zirui Tu, Zhuowen |
| contents | We provide a two-way integration for the widely adopted ControlNet by integrating external condition generation algorithms into a single dense prediction method and incorporating its individually trained image generation processes into a single model. Despite its tremendous success, the ControlNet of a two-stage pipeline bears limitations in being not self-contained (e.g. calls the external condition generation algorithms) with a large model redundancy (separately trained models for different types of conditioning inputs). Our proposed OmniControlNet consolidates 1) the condition generation (e.g., HED edges, depth maps, user scribble, and animal pose) by a single multi-tasking dense prediction algorithm under the task embedding guidance and 2) the image generation process for different conditioning types under the textual embedding guidance. OmniControlNet achieves significantly reduced model complexity and redundancy while capable of producing images of comparable quality for conditioned text-to-image generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_05871 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | OmniControlNet: Dual-stage Integration for Conditional Image Generation Wang, Yilin Xu, Haiyang Zhang, Xiang Chen, Zeyuan Sha, Zhizhou Wang, Zirui Tu, Zhuowen Computer Vision and Pattern Recognition Machine Learning We provide a two-way integration for the widely adopted ControlNet by integrating external condition generation algorithms into a single dense prediction method and incorporating its individually trained image generation processes into a single model. Despite its tremendous success, the ControlNet of a two-stage pipeline bears limitations in being not self-contained (e.g. calls the external condition generation algorithms) with a large model redundancy (separately trained models for different types of conditioning inputs). Our proposed OmniControlNet consolidates 1) the condition generation (e.g., HED edges, depth maps, user scribble, and animal pose) by a single multi-tasking dense prediction algorithm under the task embedding guidance and 2) the image generation process for different conditioning types under the textual embedding guidance. OmniControlNet achieves significantly reduced model complexity and redundancy while capable of producing images of comparable quality for conditioned text-to-image generation. |
| title | OmniControlNet: Dual-stage Integration for Conditional Image Generation |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2406.05871 |