A Mixed Diet Makes DINO An Omnivorous Vision Encoder
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917299793952768 |
|---|---|
| author | Kabra, Rishabh Ovsjanikov, Maks Hudson, Drew A. Xia, Ye Koppula, Skanda Araujo, Andre Carreira, Joao Mitra, Niloy J. |
| author_facet | Kabra, Rishabh Ovsjanikov, Maks Hudson, Drew A. Xia, Ye Koppula, Skanda Araujo, Andre Carreira, Joao Mitra, Niloy J. |
| contents | Pre-trained vision encoders like DINOv2 have demonstrated exceptional performance on unimodal tasks. However, we observe that their feature representations are poorly aligned across different modalities. For instance, the feature embedding for an RGB image and its corresponding depth map of the same scene exhibit a cosine similarity that is nearly identical to that of two random, unrelated images. To address this, we propose the Omnivorous Vision Encoder, a novel framework that learns a modality-agnostic feature space. We train the encoder with a dual objective: first, to maximize the feature alignment between different modalities of the same scene; and second, a distillation objective that anchors the learned representations to the output of a fully frozen teacher such as DINOv2. The resulting student encoder becomes "omnivorous" by producing a consistent, powerful embedding for a given scene, regardless of the input modality (RGB, Depth, Segmentation, etc.). This approach enables robust cross-modal understanding while retaining the discriminative semantics of the original foundation model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_24181 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | A Mixed Diet Makes DINO An Omnivorous Vision Encoder Kabra, Rishabh Ovsjanikov, Maks Hudson, Drew A. Xia, Ye Koppula, Skanda Araujo, Andre Carreira, Joao Mitra, Niloy J. Computer Vision and Pattern Recognition Artificial Intelligence Pre-trained vision encoders like DINOv2 have demonstrated exceptional performance on unimodal tasks. However, we observe that their feature representations are poorly aligned across different modalities. For instance, the feature embedding for an RGB image and its corresponding depth map of the same scene exhibit a cosine similarity that is nearly identical to that of two random, unrelated images. To address this, we propose the Omnivorous Vision Encoder, a novel framework that learns a modality-agnostic feature space. We train the encoder with a dual objective: first, to maximize the feature alignment between different modalities of the same scene; and second, a distillation objective that anchors the learned representations to the output of a fully frozen teacher such as DINOv2. The resulting student encoder becomes "omnivorous" by producing a consistent, powerful embedding for a given scene, regardless of the input modality (RGB, Depth, Segmentation, etc.). This approach enables robust cross-modal understanding while retaining the discriminative semantics of the original foundation model. |
| title | A Mixed Diet Makes DINO An Omnivorous Vision Encoder |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2602.24181 |