OpenVision 3: A Family of Unified Visual Encoder for Both Understanding and Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912964169170944 |
|---|---|
| author | Zhang, Letian Ren, Sucheng Liu, Yanqing Li, Xianhang Wang, Zeyu Zhou, Yuyin Yao, Huaxiu Zheng, Zeyu Nie, Weili Liu, Guilin Yu, Zhiding Xie, Cihang |
| author_facet | Zhang, Letian Ren, Sucheng Liu, Yanqing Li, Xianhang Wang, Zeyu Zhou, Yuyin Yao, Huaxiu Zheng, Zeyu Nie, Weili Liu, Guilin Yu, Zhiding Xie, Cihang |
| contents | This paper presents a family of advanced vision encoder, named OpenVision 3, that learns a single, unified visual representation that can serve both image understanding and image generation. Our core architecture is simple: we feed VAE-compressed image latents to a ViT encoder and train its output to support two complementary roles. First, the encoder output is passed to the ViT-VAE decoder to reconstruct the original image, encouraging the representation to capture generative structure. Second, the same representation is optimized with contrastive learning and image-captioning objectives, strengthening semantic features. By jointly optimizing reconstruction- and semantics-driven signals in a shared latent space, the encoder learns representations that synergize and generalize well across both regimes. We validate this unified design through extensive downstream evaluations with the encoder frozen. For generation, we test it under the RAE framework: ours substantially surpasses the standard CLIP-based encoder (e.g., gFID: 1.87 vs. 2.54 on ImageNet). For multimodal understanding, we plug the encoder into the LLaVA-1.5 and LLaVA-NeXT framework: it performs comparably with a standard CLIP vision encoder (e.g., 63.3 vs. 61.2 on SeedBench, and 59.2 vs. 58.1 on GQA). We provide empirical evidence that generation and understanding are mutually beneficial in our architecture, while further underscoring the critical role of the VAE latent space. We hope this work can spur future research on unified modeling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_15369 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | OpenVision 3: A Family of Unified Visual Encoder for Both Understanding and Generation Zhang, Letian Ren, Sucheng Liu, Yanqing Li, Xianhang Wang, Zeyu Zhou, Yuyin Yao, Huaxiu Zheng, Zeyu Nie, Weili Liu, Guilin Yu, Zhiding Xie, Cihang Image and Video Processing Artificial Intelligence This paper presents a family of advanced vision encoder, named OpenVision 3, that learns a single, unified visual representation that can serve both image understanding and image generation. Our core architecture is simple: we feed VAE-compressed image latents to a ViT encoder and train its output to support two complementary roles. First, the encoder output is passed to the ViT-VAE decoder to reconstruct the original image, encouraging the representation to capture generative structure. Second, the same representation is optimized with contrastive learning and image-captioning objectives, strengthening semantic features. By jointly optimizing reconstruction- and semantics-driven signals in a shared latent space, the encoder learns representations that synergize and generalize well across both regimes. We validate this unified design through extensive downstream evaluations with the encoder frozen. For generation, we test it under the RAE framework: ours substantially surpasses the standard CLIP-based encoder (e.g., gFID: 1.87 vs. 2.54 on ImageNet). For multimodal understanding, we plug the encoder into the LLaVA-1.5 and LLaVA-NeXT framework: it performs comparably with a standard CLIP vision encoder (e.g., 63.3 vs. 61.2 on SeedBench, and 59.2 vs. 58.1 on GQA). We provide empirical evidence that generation and understanding are mutually beneficial in our architecture, while further underscoring the critical role of the VAE latent space. We hope this work can spur future research on unified modeling. |
| title | OpenVision 3: A Family of Unified Visual Encoder for Both Understanding and Generation |
| topic | Image and Video Processing Artificial Intelligence |
| url | https://arxiv.org/abs/2601.15369 |