HelloMeme: Integrating Spatial Knitting Attentions to Embed High-Level and Fidelity-Rich Conditions in Diffusion Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914998065823744 |
|---|---|
| author | Zhang, Shengkai Jiao, Nianhong Li, Tian Yang, Chaojie Xue, Chenhui Niu, Boya Gao, Jun |
| author_facet | Zhang, Shengkai Jiao, Nianhong Li, Tian Yang, Chaojie Xue, Chenhui Niu, Boya Gao, Jun |
| contents | We propose an effective method for inserting adapters into text-to-image foundation models, which enables the execution of complex downstream tasks while preserving the generalization ability of the base model. The core idea of this method is to optimize the attention mechanism related to 2D feature maps, which enhances the performance of the adapter. This approach was validated on the task of meme video generation and achieved significant results. We hope this work can provide insights for post-training tasks of large text-to-image models. Additionally, as this method demonstrates good compatibility with SD1.5 derivative models, it holds certain value for the open-source community. Therefore, we will release the related code (\url{https://songkey.github.io/hellomeme}). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_22901 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | HelloMeme: Integrating Spatial Knitting Attentions to Embed High-Level and Fidelity-Rich Conditions in Diffusion Models Zhang, Shengkai Jiao, Nianhong Li, Tian Yang, Chaojie Xue, Chenhui Niu, Boya Gao, Jun Computer Vision and Pattern Recognition 68T07 (Primary) 68T10 I.4.5; I.5.0 We propose an effective method for inserting adapters into text-to-image foundation models, which enables the execution of complex downstream tasks while preserving the generalization ability of the base model. The core idea of this method is to optimize the attention mechanism related to 2D feature maps, which enhances the performance of the adapter. This approach was validated on the task of meme video generation and achieved significant results. We hope this work can provide insights for post-training tasks of large text-to-image models. Additionally, as this method demonstrates good compatibility with SD1.5 derivative models, it holds certain value for the open-source community. Therefore, we will release the related code (\url{https://songkey.github.io/hellomeme}). |
| title | HelloMeme: Integrating Spatial Knitting Attentions to Embed High-Level and Fidelity-Rich Conditions in Diffusion Models |
| topic | Computer Vision and Pattern Recognition 68T07 (Primary) 68T10 I.4.5; I.5.0 |
| url | https://arxiv.org/abs/2410.22901 |