RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Control
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866907914811211776 |
|---|---|
| author | Rout, Litu Chen, Yujia Ruiz, Nataniel Kumar, Abhishek Caramanis, Constantine Shakkottai, Sanjay Chu, Wen-Sheng |
| author_facet | Rout, Litu Chen, Yujia Ruiz, Nataniel Kumar, Abhishek Caramanis, Constantine Shakkottai, Sanjay Chu, Wen-Sheng |
| contents | We propose Reference-Based Modulation (RB-Modulation), a new plug-and-play solution for training-free personalization of diffusion models. Existing training-free approaches exhibit difficulties in (a) style extraction from reference images in the absence of additional style or content text descriptions, (b) unwanted content leakage from reference style images, and (c) effective composition of style and content. RB-Modulation is built on a novel stochastic optimal controller where a style descriptor encodes the desired attributes through a terminal cost. The resulting drift not only overcomes the difficulties above, but also ensures high fidelity to the reference style and adheres to the given text prompt. We also introduce a cross-attention-based feature aggregation scheme that allows RB-Modulation to decouple content and style from the reference image. With theoretical justification and empirical evidence, our framework demonstrates precise extraction and control of content and style in a training-free manner. Further, our method allows a seamless composition of content and style, which marks a departure from the dependency on external adapters or ControlNets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_17401 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Control Rout, Litu Chen, Yujia Ruiz, Nataniel Kumar, Abhishek Caramanis, Constantine Shakkottai, Sanjay Chu, Wen-Sheng Machine Learning Computer Vision and Pattern Recognition We propose Reference-Based Modulation (RB-Modulation), a new plug-and-play solution for training-free personalization of diffusion models. Existing training-free approaches exhibit difficulties in (a) style extraction from reference images in the absence of additional style or content text descriptions, (b) unwanted content leakage from reference style images, and (c) effective composition of style and content. RB-Modulation is built on a novel stochastic optimal controller where a style descriptor encodes the desired attributes through a terminal cost. The resulting drift not only overcomes the difficulties above, but also ensures high fidelity to the reference style and adheres to the given text prompt. We also introduce a cross-attention-based feature aggregation scheme that allows RB-Modulation to decouple content and style from the reference image. With theoretical justification and empirical evidence, our framework demonstrates precise extraction and control of content and style in a training-free manner. Further, our method allows a seamless composition of content and style, which marks a departure from the dependency on external adapters or ControlNets. |
| title | RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Control |
| topic | Machine Learning Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2405.17401 |