Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2412.01027 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913595245199360 |
|---|---|
| author | Lai, Bolin Juefei-Xu, Felix Liu, Miao Dai, Xiaoliang Mehta, Nikhil Zhu, Chenguang Huang, Zeyi Rehg, James M. Lee, Sangmin Zhang, Ning Xiao, Tong |
| author_facet | Lai, Bolin Juefei-Xu, Felix Liu, Miao Dai, Xiaoliang Mehta, Nikhil Zhu, Chenguang Huang, Zeyi Rehg, James M. Lee, Sangmin Zhang, Ning Xiao, Tong |
| contents | Text-guided image manipulation has experienced notable advancement in recent years. In order to mitigate linguistic ambiguity, few-shot learning with visual examples has been applied for instructions that are underrepresented in the training set, or difficult to describe purely in language. However, learning from visual prompts requires strong reasoning capability, which diffusion models are struggling with. To address this issue, we introduce a novel multi-modal autoregressive model, dubbed $\textbf{InstaManip}$, that can $\textbf{insta}$ntly learn a new image $\textbf{manip}$ulation operation from textual and visual guidance via in-context learning, and apply it to new query images. Specifically, we propose an innovative group self-attention mechanism to break down the in-context learning process into two separate stages -- learning and applying, which simplifies the complex problem into two easier tasks. We also introduce a relation regularization method to further disentangle image transformation features from irrelevant contents in exemplar images. Extensive experiments suggest that our method surpasses previous few-shot image manipulation models by a notable margin ($\geq$19% in human evaluation). We also find our model can be further boosted by increasing the number or diversity of exemplar images. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_01027 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation Lai, Bolin Juefei-Xu, Felix Liu, Miao Dai, Xiaoliang Mehta, Nikhil Zhu, Chenguang Huang, Zeyi Rehg, James M. Lee, Sangmin Zhang, Ning Xiao, Tong Computer Vision and Pattern Recognition Text-guided image manipulation has experienced notable advancement in recent years. In order to mitigate linguistic ambiguity, few-shot learning with visual examples has been applied for instructions that are underrepresented in the training set, or difficult to describe purely in language. However, learning from visual prompts requires strong reasoning capability, which diffusion models are struggling with. To address this issue, we introduce a novel multi-modal autoregressive model, dubbed $\textbf{InstaManip}$, that can $\textbf{insta}$ntly learn a new image $\textbf{manip}$ulation operation from textual and visual guidance via in-context learning, and apply it to new query images. Specifically, we propose an innovative group self-attention mechanism to break down the in-context learning process into two separate stages -- learning and applying, which simplifies the complex problem into two easier tasks. We also introduce a relation regularization method to further disentangle image transformation features from irrelevant contents in exemplar images. Extensive experiments suggest that our method surpasses previous few-shot image manipulation models by a notable margin ($\geq$19% in human evaluation). We also find our model can be further boosted by increasing the number or diversity of exemplar images. |
| title | Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.01027 |