SubZero: Composing Subject, Style, and Action via Zero-Shot Personalization
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913710119845888 |
|---|---|
| author | Borse, Shubhankar Bhardwaj, Kartikeya Dastjerdi, Mohammad Reza Karimi Park, Hyojin Kadambi, Shreya Shivakumar, Shobitha Mandke, Prathamesh Nayak, Ankita Teague, Harris Hayat, Munawar Porikli, Fatih |
| author_facet | Borse, Shubhankar Bhardwaj, Kartikeya Dastjerdi, Mohammad Reza Karimi Park, Hyojin Kadambi, Shreya Shivakumar, Shobitha Mandke, Prathamesh Nayak, Ankita Teague, Harris Hayat, Munawar Porikli, Fatih |
| contents | Diffusion models are increasingly popular for generative tasks, including personalized composition of subjects and styles. While diffusion models can generate user-specified subjects performing text-guided actions in custom styles, they require fine-tuning and are not feasible for personalization on mobile devices. Hence, tuning-free personalization methods such as IP-Adapters have progressively gained traction. However, for the composition of subjects and styles, these works are less flexible due to their reliance on ControlNet, or show content and style leakage artifacts. To tackle these, we present SubZero, a novel framework to generate any subject in any style, performing any action without the need for fine-tuning. We propose a novel set of constraints to enhance subject and style similarity, while reducing leakage. Additionally, we propose an orthogonalized temporal aggregation scheme in the cross-attention blocks of denoising model, effectively conditioning on a text prompt along with single subject and style images. We also propose a novel method to train customized content and style projectors to reduce content and style leakage. Through extensive experiments, we show that our proposed approach, while suitable for running on-edge, shows significant improvements over state-of-the-art works performing subject, style and action composition. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_19673 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SubZero: Composing Subject, Style, and Action via Zero-Shot Personalization Borse, Shubhankar Bhardwaj, Kartikeya Dastjerdi, Mohammad Reza Karimi Park, Hyojin Kadambi, Shreya Shivakumar, Shobitha Mandke, Prathamesh Nayak, Ankita Teague, Harris Hayat, Munawar Porikli, Fatih Computer Vision and Pattern Recognition Diffusion models are increasingly popular for generative tasks, including personalized composition of subjects and styles. While diffusion models can generate user-specified subjects performing text-guided actions in custom styles, they require fine-tuning and are not feasible for personalization on mobile devices. Hence, tuning-free personalization methods such as IP-Adapters have progressively gained traction. However, for the composition of subjects and styles, these works are less flexible due to their reliance on ControlNet, or show content and style leakage artifacts. To tackle these, we present SubZero, a novel framework to generate any subject in any style, performing any action without the need for fine-tuning. We propose a novel set of constraints to enhance subject and style similarity, while reducing leakage. Additionally, we propose an orthogonalized temporal aggregation scheme in the cross-attention blocks of denoising model, effectively conditioning on a text prompt along with single subject and style images. We also propose a novel method to train customized content and style projectors to reduce content and style leakage. Through extensive experiments, we show that our proposed approach, while suitable for running on-edge, shows significant improvements over state-of-the-art works performing subject, style and action composition. |
| title | SubZero: Composing Subject, Style, and Action via Zero-Shot Personalization |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2502.19673 |