Exploring MLLM-Diffusion Information Transfer with MetaCanvas
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911315787776000 |
|---|---|
| author | Lin, Han Pan, Xichen Huang, Ziqi Hou, Ji Wang, Jialiang Chen, Weifeng He, Zecheng Juefei-Xu, Felix Sun, Junzhe Fan, Zhipeng Thabet, Ali Bansal, Mohit Wang, Chu |
| author_facet | Lin, Han Pan, Xichen Huang, Ziqi Hou, Ji Wang, Jialiang Chen, Weifeng He, Zecheng Juefei-Xu, Felix Sun, Junzhe Fan, Zhipeng Thabet, Ali Bansal, Mohit Wang, Chu |
| contents | Multimodal learning has rapidly advanced visual understanding, largely via multimodal large language models (MLLMs) that use powerful LLMs as cognitive cores. In visual generation, however, these powerful core models are typically reduced to global text encoders for diffusion models, leaving most of their reasoning and planning ability unused. This creates a gap: current multimodal LLMs can parse complex layouts, attributes, and knowledge-intensive scenes, yet struggle to generate images or videos with equally precise and structured control. We propose MetaCanvas, a lightweight framework that lets MLLMs reason and plan directly in spatial and spatiotemporal latent spaces and interface tightly with diffusion generators. We empirically implement MetaCanvas on three different diffusion backbones and evaluate it across six tasks, including text-to-image generation, text/image-to-video generation, image/video editing, and in-context video generation, each requiring precise layouts, robust attribute binding, and reasoning-intensive control. MetaCanvas consistently outperforms global-conditioning baselines, suggesting that treating MLLMs as latent-space planners is a promising direction for narrowing the gap between multimodal understanding and generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_11464 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Exploring MLLM-Diffusion Information Transfer with MetaCanvas Lin, Han Pan, Xichen Huang, Ziqi Hou, Ji Wang, Jialiang Chen, Weifeng He, Zecheng Juefei-Xu, Felix Sun, Junzhe Fan, Zhipeng Thabet, Ali Bansal, Mohit Wang, Chu Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Multimodal learning has rapidly advanced visual understanding, largely via multimodal large language models (MLLMs) that use powerful LLMs as cognitive cores. In visual generation, however, these powerful core models are typically reduced to global text encoders for diffusion models, leaving most of their reasoning and planning ability unused. This creates a gap: current multimodal LLMs can parse complex layouts, attributes, and knowledge-intensive scenes, yet struggle to generate images or videos with equally precise and structured control. We propose MetaCanvas, a lightweight framework that lets MLLMs reason and plan directly in spatial and spatiotemporal latent spaces and interface tightly with diffusion generators. We empirically implement MetaCanvas on three different diffusion backbones and evaluate it across six tasks, including text-to-image generation, text/image-to-video generation, image/video editing, and in-context video generation, each requiring precise layouts, robust attribute binding, and reasoning-intensive control. MetaCanvas consistently outperforms global-conditioning baselines, suggesting that treating MLLMs as latent-space planners is a promising direction for narrowing the gap between multimodal understanding and generation. |
| title | Exploring MLLM-Diffusion Information Transfer with MetaCanvas |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2512.11464 |