Proxy-Tuning: Tailoring Multimodal Autoregressive Models for Subject-Driven Image Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918223237087232 |
|---|---|
| author | Wu, Yi Qian, Shengju Zhu, Lingting Liu, Lei Qiao, Wandi Li, Ziqiang Yu, Lequan Li, Bin |
| author_facet | Wu, Yi Qian, Shengju Zhu, Lingting Liu, Lei Qiao, Wandi Li, Ziqiang Yu, Lequan Li, Bin |
| contents | Multimodal autoregressive (AR) models, based on next-token prediction and transformer architecture, have demonstrated remarkable capabilities in various multimodal tasks including text-to-image (T2I) generation. Despite their strong performance in general T2I tasks, our research reveals that these models initially struggle with subject-driven image generation compared to dominant diffusion models. To address this limitation, we introduce Proxy-Tuning, leveraging diffusion models to enhance AR models' capabilities in subject-specific image generation. Our method reveals a striking weak-to-strong phenomenon: fine-tuned AR models consistently outperform their diffusion model supervisors in both subject fidelity and prompt adherence. We analyze this performance shift and identify scenarios where AR models excel, particularly in multi-subject compositions and contextual understanding. This work not only demonstrates impressive results in subject-driven AR image generation, but also unveils the potential of weak-to-strong generalization in the image generation domain, contributing to a deeper understanding of different architectures' strengths and limitations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_10125 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Proxy-Tuning: Tailoring Multimodal Autoregressive Models for Subject-Driven Image Generation Wu, Yi Qian, Shengju Zhu, Lingting Liu, Lei Qiao, Wandi Li, Ziqiang Yu, Lequan Li, Bin Computer Vision and Pattern Recognition Multimedia Multimodal autoregressive (AR) models, based on next-token prediction and transformer architecture, have demonstrated remarkable capabilities in various multimodal tasks including text-to-image (T2I) generation. Despite their strong performance in general T2I tasks, our research reveals that these models initially struggle with subject-driven image generation compared to dominant diffusion models. To address this limitation, we introduce Proxy-Tuning, leveraging diffusion models to enhance AR models' capabilities in subject-specific image generation. Our method reveals a striking weak-to-strong phenomenon: fine-tuned AR models consistently outperform their diffusion model supervisors in both subject fidelity and prompt adherence. We analyze this performance shift and identify scenarios where AR models excel, particularly in multi-subject compositions and contextual understanding. This work not only demonstrates impressive results in subject-driven AR image generation, but also unveils the potential of weak-to-strong generalization in the image generation domain, contributing to a deeper understanding of different architectures' strengths and limitations. |
| title | Proxy-Tuning: Tailoring Multimodal Autoregressive Models for Subject-Driven Image Generation |
| topic | Computer Vision and Pattern Recognition Multimedia |
| url | https://arxiv.org/abs/2503.10125 |