Text-to-Sticker: Style Tailoring Latent Diffusion Models for Human Expression
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909333338456064 |
|---|---|
| author | Sinha, Animesh Sun, Bo Kalia, Anmol Casanova, Arantxa Blanchard, Elliot Yan, David Zhang, Winnie Nelli, Tony Chen, Jiahui Shah, Hardik Yu, Licheng Singh, Mitesh Kumar Ramchandani, Ankit Sanjabi, Maziar Gupta, Sonal Bearman, Amy Mahajan, Dhruv |
| author_facet | Sinha, Animesh Sun, Bo Kalia, Anmol Casanova, Arantxa Blanchard, Elliot Yan, David Zhang, Winnie Nelli, Tony Chen, Jiahui Shah, Hardik Yu, Licheng Singh, Mitesh Kumar Ramchandani, Ankit Sanjabi, Maziar Gupta, Sonal Bearman, Amy Mahajan, Dhruv |
| contents | We introduce Style Tailoring, a recipe to finetune Latent Diffusion Models (LDMs) in a distinct domain with high visual quality, prompt alignment and scene diversity. We choose sticker image generation as the target domain, as the images significantly differ from photorealistic samples typically generated by large-scale LDMs. We start with a competent text-to-image model, like Emu, and show that relying on prompt engineering with a photorealistic model to generate stickers leads to poor prompt alignment and scene diversity. To overcome these drawbacks, we first finetune Emu on millions of sticker-like images collected using weak supervision to elicit diversity. Next, we curate human-in-the-loop (HITL) Alignment and Style datasets from model generations, and finetune to improve prompt alignment and style alignment respectively. Sequential finetuning on these datasets poses a tradeoff between better style alignment and prompt alignment gains. To address this tradeoff, we propose a novel fine-tuning method called Style Tailoring, which jointly fits the content and style distribution and achieves best tradeoff. Evaluation results show our method improves visual quality by 14%, prompt alignment by 16.2% and scene diversity by 15.3%, compared to prompt engineering the base Emu model for stickers generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_10794 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Text-to-Sticker: Style Tailoring Latent Diffusion Models for Human Expression Sinha, Animesh Sun, Bo Kalia, Anmol Casanova, Arantxa Blanchard, Elliot Yan, David Zhang, Winnie Nelli, Tony Chen, Jiahui Shah, Hardik Yu, Licheng Singh, Mitesh Kumar Ramchandani, Ankit Sanjabi, Maziar Gupta, Sonal Bearman, Amy Mahajan, Dhruv Computer Vision and Pattern Recognition We introduce Style Tailoring, a recipe to finetune Latent Diffusion Models (LDMs) in a distinct domain with high visual quality, prompt alignment and scene diversity. We choose sticker image generation as the target domain, as the images significantly differ from photorealistic samples typically generated by large-scale LDMs. We start with a competent text-to-image model, like Emu, and show that relying on prompt engineering with a photorealistic model to generate stickers leads to poor prompt alignment and scene diversity. To overcome these drawbacks, we first finetune Emu on millions of sticker-like images collected using weak supervision to elicit diversity. Next, we curate human-in-the-loop (HITL) Alignment and Style datasets from model generations, and finetune to improve prompt alignment and style alignment respectively. Sequential finetuning on these datasets poses a tradeoff between better style alignment and prompt alignment gains. To address this tradeoff, we propose a novel fine-tuning method called Style Tailoring, which jointly fits the content and style distribution and achieves best tradeoff. Evaluation results show our method improves visual quality by 14%, prompt alignment by 16.2% and scene diversity by 15.3%, compared to prompt engineering the base Emu model for stickers generation. |
| title | Text-to-Sticker: Style Tailoring Latent Diffusion Models for Human Expression |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2311.10794 |