Training-Free Style and Content Transfer by Leveraging U-Net Skip Connections in Stable Diffusion
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913775921135616 |
|---|---|
| author | Schaerf, Ludovica Alfarano, Andrea Silvestri, Fabrizio Impett, Leonardo |
| author_facet | Schaerf, Ludovica Alfarano, Andrea Silvestri, Fabrizio Impett, Leonardo |
| contents | Recent advances in diffusion models for image generation have led to detailed examinations of several components within the U-Net architecture for image editing. While previous studies have focused on the bottleneck layer (h-space), cross-attention, self-attention, and decoding layers, the overall role of the skip connections of the U-Net itself has not been specifically addressed. We conduct thorough analyses on the role of the skip connections and find that the residual connections passed by the third encoder block carry most of the spatial information of the reconstructed image, splitting the content from the style, passed by the remaining stream in the opposed decoding layer. We show that injecting the representations from this block can be used for text-based editing, precise modifications, and style transfer. We compare our method, SkipInject, to state-of-the-art style transfer and image editing methods and demonstrate that our method obtains the best content alignment and optimal structural preservation tradeoff. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_14524 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Training-Free Style and Content Transfer by Leveraging U-Net Skip Connections in Stable Diffusion Schaerf, Ludovica Alfarano, Andrea Silvestri, Fabrizio Impett, Leonardo Computer Vision and Pattern Recognition Recent advances in diffusion models for image generation have led to detailed examinations of several components within the U-Net architecture for image editing. While previous studies have focused on the bottleneck layer (h-space), cross-attention, self-attention, and decoding layers, the overall role of the skip connections of the U-Net itself has not been specifically addressed. We conduct thorough analyses on the role of the skip connections and find that the residual connections passed by the third encoder block carry most of the spatial information of the reconstructed image, splitting the content from the style, passed by the remaining stream in the opposed decoding layer. We show that injecting the representations from this block can be used for text-based editing, precise modifications, and style transfer. We compare our method, SkipInject, to state-of-the-art style transfer and image editing methods and demonstrate that our method obtains the best content alignment and optimal structural preservation tradeoff. |
| title | Training-Free Style and Content Transfer by Leveraging U-Net Skip Connections in Stable Diffusion |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2501.14524 |