DuoGen: Towards General Purpose Interleaved Multimodal Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918320404430848 |
|---|---|
| author | Shi, Min Zeng, Xiaohui Huang, Jiannan Cui, Yin Ferroni, Francesco Li, Jialuo Pachori, Shubham Li, Zhaoshuo Balaji, Yogesh Wang, Haoxiang Lin, Tsung-Yi Fu, Xiao Zhao, Yue Chen, Chieh-Yun Liu, Ming-Yu Shi, Humphrey |
| author_facet | Shi, Min Zeng, Xiaohui Huang, Jiannan Cui, Yin Ferroni, Francesco Li, Jialuo Pachori, Shubham Li, Zhaoshuo Balaji, Yogesh Wang, Haoxiang Lin, Tsung-Yi Fu, Xiao Zhao, Yue Chen, Chieh-Yun Liu, Ming-Yu Shi, Humphrey |
| contents | Interleaved multimodal generation enables capabilities beyond unimodal generation models, such as step-by-step instructional guides, visual planning, and generating visual drafts for reasoning. However, the quality of existing interleaved generation models under general instructions remains limited by insufficient training data and base model capacity. We present DuoGen, a general-purpose interleaved generation framework that systematically addresses data curation, architecture design, and evaluation. On the data side, we build a large-scale, high-quality instruction-tuning dataset by combining multimodal conversations rewritten from curated raw websites, and diverse synthetic examples covering everyday scenarios. Architecturally, DuoGen leverages the strong visual understanding of a pretrained multimodal LLM and the visual generation capabilities of a diffusion transformer (DiT) pretrained on video generation, avoiding costly unimodal pretraining and enabling flexible base model selection. A two-stage decoupled strategy first instruction-tunes the MLLM, then aligns DiT with it using curated interleaved image-text sequences. Across public and newly proposed benchmarks, DuoGen outperforms prior open-source models in text quality, image fidelity, and image-context alignment, and also achieves state-of-the-art performance on text-to-image and image editing among unified generation models. Data and code will be released at https://research.nvidia.com/labs/dir/duogen/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_00508 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | DuoGen: Towards General Purpose Interleaved Multimodal Generation Shi, Min Zeng, Xiaohui Huang, Jiannan Cui, Yin Ferroni, Francesco Li, Jialuo Pachori, Shubham Li, Zhaoshuo Balaji, Yogesh Wang, Haoxiang Lin, Tsung-Yi Fu, Xiao Zhao, Yue Chen, Chieh-Yun Liu, Ming-Yu Shi, Humphrey Computer Vision and Pattern Recognition Interleaved multimodal generation enables capabilities beyond unimodal generation models, such as step-by-step instructional guides, visual planning, and generating visual drafts for reasoning. However, the quality of existing interleaved generation models under general instructions remains limited by insufficient training data and base model capacity. We present DuoGen, a general-purpose interleaved generation framework that systematically addresses data curation, architecture design, and evaluation. On the data side, we build a large-scale, high-quality instruction-tuning dataset by combining multimodal conversations rewritten from curated raw websites, and diverse synthetic examples covering everyday scenarios. Architecturally, DuoGen leverages the strong visual understanding of a pretrained multimodal LLM and the visual generation capabilities of a diffusion transformer (DiT) pretrained on video generation, avoiding costly unimodal pretraining and enabling flexible base model selection. A two-stage decoupled strategy first instruction-tunes the MLLM, then aligns DiT with it using curated interleaved image-text sequences. Across public and newly proposed benchmarks, DuoGen outperforms prior open-source models in text quality, image fidelity, and image-context alignment, and also achieves state-of-the-art performance on text-to-image and image editing among unified generation models. Data and code will be released at https://research.nvidia.com/labs/dir/duogen/. |
| title | DuoGen: Towards General Purpose Interleaved Multimodal Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2602.00508 |