Unifying Contrastive and Generative Objectives for Visual Understanding and Text-to-Image Generation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914573802536960 |
|---|---|
| author | Li, Chao Li, Tianhong Nuthalapati, Sai Vidyaranya Chen, Hong-You Shukla, Satya Narayan Cheng, Jianpeng Yang, Yonghuan Xiao, Jun Fan, Xiangjun Singh, Aashu Katabi, Dina Mishra, Shlok Kumar |
| author_facet | Li, Chao Li, Tianhong Nuthalapati, Sai Vidyaranya Chen, Hong-You Shukla, Satya Narayan Cheng, Jianpeng Yang, Yonghuan Xiao, Jun Fan, Xiangjun Singh, Aashu Katabi, Dina Mishra, Shlok Kumar |
| contents | Unifying text-image contrastive learning and text-to-image (T2I) generation in a single end-to-end model is challenging because the two objectives demand opposing masking regimes: contrastive alignment needs near-complete visible tokens, while masked generative modeling needs heavy corruption. We introduce DREAM, a unified framework that resolves this conflict through Masking Warmup, a schedule that shifts the center of the masking distribution over training, so low and high masking ratios coexist at every step. This co-exposure lets a single jointly-trained encoder serve both objectives. The resulting stable optimization unlocks Semantically Aligned Decoding at inference: the text encoder, trained against visual embeddings at all masking ratios, can score partially generated images and select the best trajectory with as little as 12.5% of the image decoded, improving both FID and throughput. DREAM outperforms its single-objective baselines, CLIP and FLUID: on ImageNet linear-probing (+1.1%), 5-shot transfer (+4.1%), ADE20K segmentation (+1.9%), and NYU depth estimation (+6.25%) over CLIP, and on CC12M FID (+6.2%) over FLUID while maintaining CLIP Score. Together, these gains show that text-image contrastive and generative objectives, when properly unified, are synergistic rather than competing. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_02667 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Unifying Contrastive and Generative Objectives for Visual Understanding and Text-to-Image Generation Li, Chao Li, Tianhong Nuthalapati, Sai Vidyaranya Chen, Hong-You Shukla, Satya Narayan Cheng, Jianpeng Yang, Yonghuan Xiao, Jun Fan, Xiangjun Singh, Aashu Katabi, Dina Mishra, Shlok Kumar Computer Vision and Pattern Recognition Machine Learning Unifying text-image contrastive learning and text-to-image (T2I) generation in a single end-to-end model is challenging because the two objectives demand opposing masking regimes: contrastive alignment needs near-complete visible tokens, while masked generative modeling needs heavy corruption. We introduce DREAM, a unified framework that resolves this conflict through Masking Warmup, a schedule that shifts the center of the masking distribution over training, so low and high masking ratios coexist at every step. This co-exposure lets a single jointly-trained encoder serve both objectives. The resulting stable optimization unlocks Semantically Aligned Decoding at inference: the text encoder, trained against visual embeddings at all masking ratios, can score partially generated images and select the best trajectory with as little as 12.5% of the image decoded, improving both FID and throughput. DREAM outperforms its single-objective baselines, CLIP and FLUID: on ImageNet linear-probing (+1.1%), 5-shot transfer (+4.1%), ADE20K segmentation (+1.9%), and NYU depth estimation (+6.25%) over CLIP, and on CC12M FID (+6.2%) over FLUID while maintaining CLIP Score. Together, these gains show that text-image contrastive and generative objectives, when properly unified, are synergistic rather than competing. |
| title | Unifying Contrastive and Generative Objectives for Visual Understanding and Text-to-Image Generation |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2603.02667 |