LCM-Lookahead for Encoder-based Text-to-Image Personalization
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914741207695360 |
|---|---|
| author | Gal, Rinon Lichter, Or Richardson, Elad Patashnik, Or Bermano, Amit H. Chechik, Gal Cohen-Or, Daniel |
| author_facet | Gal, Rinon Lichter, Or Richardson, Elad Patashnik, Or Bermano, Amit H. Chechik, Gal Cohen-Or, Daniel |
| contents | Recent advancements in diffusion models have introduced fast sampling methods that can effectively produce high-quality images in just one or a few denoising steps. Interestingly, when these are distilled from existing diffusion models, they often maintain alignment with the original model, retaining similar outputs for similar prompts and seeds. These properties present opportunities to leverage fast sampling methods as a shortcut-mechanism, using them to create a preview of denoised outputs through which we can backpropagate image-space losses. In this work, we explore the potential of using such shortcut-mechanisms to guide the personalization of text-to-image models to specific facial identities. We focus on encoder-based personalization approaches, and demonstrate that by tuning them with a lookahead identity loss, we can achieve higher identity fidelity, without sacrificing layout diversity or prompt alignment. We further explore the use of attention sharing mechanisms and consistent data generation for the task of personalization, and find that encoder training can benefit from both. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_03620 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LCM-Lookahead for Encoder-based Text-to-Image Personalization Gal, Rinon Lichter, Or Richardson, Elad Patashnik, Or Bermano, Amit H. Chechik, Gal Cohen-Or, Daniel Computer Vision and Pattern Recognition Graphics Recent advancements in diffusion models have introduced fast sampling methods that can effectively produce high-quality images in just one or a few denoising steps. Interestingly, when these are distilled from existing diffusion models, they often maintain alignment with the original model, retaining similar outputs for similar prompts and seeds. These properties present opportunities to leverage fast sampling methods as a shortcut-mechanism, using them to create a preview of denoised outputs through which we can backpropagate image-space losses. In this work, we explore the potential of using such shortcut-mechanisms to guide the personalization of text-to-image models to specific facial identities. We focus on encoder-based personalization approaches, and demonstrate that by tuning them with a lookahead identity loss, we can achieve higher identity fidelity, without sacrificing layout diversity or prompt alignment. We further explore the use of attention sharing mechanisms and consistent data generation for the task of personalization, and find that encoder training can benefit from both. |
| title | LCM-Lookahead for Encoder-based Text-to-Image Personalization |
| topic | Computer Vision and Pattern Recognition Graphics |
| url | https://arxiv.org/abs/2404.03620 |