Transfer between Modalities with MetaQueries
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912316691316736 |
|---|---|
| author | Pan, Xichen Shukla, Satya Narayan Singh, Aashu Zhao, Zhuokai Mishra, Shlok Kumar Wang, Jialiang Xu, Zhiyang Chen, Jiuhai Li, Kunpeng Juefei-Xu, Felix Hou, Ji Xie, Saining |
| author_facet | Pan, Xichen Shukla, Satya Narayan Singh, Aashu Zhao, Zhuokai Mishra, Shlok Kumar Wang, Jialiang Xu, Zhiyang Chen, Jiuhai Li, Kunpeng Juefei-Xu, Felix Hou, Ji Xie, Saining |
| contents | Unified multimodal models aim to integrate understanding (text output) and generation (pixel output), but aligning these different modalities within a single architecture often demands complex training recipes and careful data balancing. We introduce MetaQueries, a set of learnable queries that act as an efficient interface between autoregressive multimodal LLMs (MLLMs) and diffusion models. MetaQueries connects the MLLM's latents to the diffusion decoder, enabling knowledge-augmented image generation by leveraging the MLLM's deep understanding and reasoning capabilities. Our method simplifies training, requiring only paired image-caption data and standard diffusion objectives. Notably, this transfer is effective even when the MLLM backbone remains frozen, thereby preserving its state-of-the-art multimodal understanding capabilities while achieving strong generative performance. Additionally, our method is flexible and can be easily instruction-tuned for advanced applications such as image editing and subject-driven generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_06256 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Transfer between Modalities with MetaQueries Pan, Xichen Shukla, Satya Narayan Singh, Aashu Zhao, Zhuokai Mishra, Shlok Kumar Wang, Jialiang Xu, Zhiyang Chen, Jiuhai Li, Kunpeng Juefei-Xu, Felix Hou, Ji Xie, Saining Computer Vision and Pattern Recognition Unified multimodal models aim to integrate understanding (text output) and generation (pixel output), but aligning these different modalities within a single architecture often demands complex training recipes and careful data balancing. We introduce MetaQueries, a set of learnable queries that act as an efficient interface between autoregressive multimodal LLMs (MLLMs) and diffusion models. MetaQueries connects the MLLM's latents to the diffusion decoder, enabling knowledge-augmented image generation by leveraging the MLLM's deep understanding and reasoning capabilities. Our method simplifies training, requiring only paired image-caption data and standard diffusion objectives. Notably, this transfer is effective even when the MLLM backbone remains frozen, thereby preserving its state-of-the-art multimodal understanding capabilities while achieving strong generative performance. Additionally, our method is flexible and can be easily instruction-tuned for advanced applications such as image editing and subject-driven generation. |
| title | Transfer between Modalities with MetaQueries |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2504.06256 |