OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910033419173888 |
|---|---|
| author | Tang, Feilong An, Xiang Yan, Yunyao Xie, Yin Qin, Bin Yang, Kaicheng Shen, Yifei Zhang, Yuanhan Li, Chunyuan Feng, Shikun Chen, Changrui Tan, Huajie Hu, Ming Zhang, Manyuan Li, Bo Feng, Ziyong Liu, Ziwei Ge, Zongyuan Deng, Jiankang |
| author_facet | Tang, Feilong An, Xiang Yan, Yunyao Xie, Yin Qin, Bin Yang, Kaicheng Shen, Yifei Zhang, Yuanhan Li, Chunyuan Feng, Shikun Chen, Changrui Tan, Huajie Hu, Ming Zhang, Manyuan Li, Bo Feng, Ziyong Liu, Ziwei Ge, Zongyuan Deng, Jiankang |
| contents | Hypothesis. Artificial general intelligence is, at its core, a compression problem. Effective compression demands resonance: deep learning scales best when its architecture aligns with the fundamental structure of the data. These are the fundamental principles. Yet, modern vision architectures have strayed from these truths: visual signals are highly redundant, while discriminative information, the surprise, is sparse. Current models process dense pixel grids uniformly, wasting vast compute on static background rather than focusing on the predictive residuals that define motion and meaning. We argue that to solve visual understanding, we must align our architectures with the information-theoretic principles of video, i.e., Codecs.
Method. OneVision-Encoder encodes video by compressing predictive visual structure into semantic meaning. By adopting Codec Patchification, OV-Encoder abandons uniform computation to focus exclusively on the 3.1%-25% of regions rich in signal entropy. To unify spatial and temporal reasoning under irregular token layouts, OneVision-Encoder employs a shared 3D RoPE and is trained with a large-scale cluster discrimination objective over more than one million semantic concepts, jointly capturing object permanence and motion dynamics.
Evidence. The results validate our core hypothesis: efficiency and accuracy are not a trade-off; they are positively correlated. When integrated into LLM, it consistently outperforms strong vision backbones such as Qwen3-ViT and SigLIP2 across 16 image, video, and document understanding benchmarks, despite using substantially fewer visual tokens and pretraining data. Notably, on video understanding tasks, OV-Encoder achieves an average improvement of 4.1% over Qwen3-ViT. Codec-aligned, patch-level sparsity is a foundational principle, enabling OV-Encoder as a scalable engine for next-generation visual generalists. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_08683 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence Tang, Feilong An, Xiang Yan, Yunyao Xie, Yin Qin, Bin Yang, Kaicheng Shen, Yifei Zhang, Yuanhan Li, Chunyuan Feng, Shikun Chen, Changrui Tan, Huajie Hu, Ming Zhang, Manyuan Li, Bo Feng, Ziyong Liu, Ziwei Ge, Zongyuan Deng, Jiankang Computer Vision and Pattern Recognition Hypothesis. Artificial general intelligence is, at its core, a compression problem. Effective compression demands resonance: deep learning scales best when its architecture aligns with the fundamental structure of the data. These are the fundamental principles. Yet, modern vision architectures have strayed from these truths: visual signals are highly redundant, while discriminative information, the surprise, is sparse. Current models process dense pixel grids uniformly, wasting vast compute on static background rather than focusing on the predictive residuals that define motion and meaning. We argue that to solve visual understanding, we must align our architectures with the information-theoretic principles of video, i.e., Codecs. Method. OneVision-Encoder encodes video by compressing predictive visual structure into semantic meaning. By adopting Codec Patchification, OV-Encoder abandons uniform computation to focus exclusively on the 3.1%-25% of regions rich in signal entropy. To unify spatial and temporal reasoning under irregular token layouts, OneVision-Encoder employs a shared 3D RoPE and is trained with a large-scale cluster discrimination objective over more than one million semantic concepts, jointly capturing object permanence and motion dynamics. Evidence. The results validate our core hypothesis: efficiency and accuracy are not a trade-off; they are positively correlated. When integrated into LLM, it consistently outperforms strong vision backbones such as Qwen3-ViT and SigLIP2 across 16 image, video, and document understanding benchmarks, despite using substantially fewer visual tokens and pretraining data. Notably, on video understanding tasks, OV-Encoder achieves an average improvement of 4.1% over Qwen3-ViT. Codec-aligned, patch-level sparsity is a foundational principle, enabling OV-Encoder as a scalable engine for next-generation visual generalists. |
| title | OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2602.08683 |