Scope: Selective Cross-modal Orchestration of Visual Perception Experts
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909851920105472 |
|---|---|
| author | Zhang, Tianyu Wang, Suyuchen Wang, Chao Rodriguez, Juan Masry, Ahmed Jian, Xiangru Bengio, Yoshua Taslakian, Perouz |
| author_facet | Zhang, Tianyu Wang, Suyuchen Wang, Chao Rodriguez, Juan Masry, Ahmed Jian, Xiangru Bengio, Yoshua Taslakian, Perouz |
| contents | Vision-language models (VLMs) benefit from multiple vision encoders, but naively stacking them yields diminishing returns while multiplying inference costs. We propose SCOPE, a Mixture-of-Encoders (MoEnc) framework that dynamically selects one specialized encoder per image-text pair via instance-level routing, unlike token-level routing in traditional MoE. SCOPE maintains a shared encoder and a pool of routed encoders. A lightweight router uses cross-attention between text prompts and shared visual features to select the optimal encoder from the routed encoders. To train this router, we introduce dual entropy regularization with auxiliary losses to balance dataset-level load distribution with instance-level routing confidence. Remarkably, SCOPE with one shared plus one routed encoder outperforms models using all four extra encoders simultaneously, while reducing compute by 24-49\%. This demonstrates that intelligent encoder selection beats brute-force aggregation, challenging the prevailing paradigm in multi-encoder VLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_12974 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scope: Selective Cross-modal Orchestration of Visual Perception Experts Zhang, Tianyu Wang, Suyuchen Wang, Chao Rodriguez, Juan Masry, Ahmed Jian, Xiangru Bengio, Yoshua Taslakian, Perouz Computer Vision and Pattern Recognition Vision-language models (VLMs) benefit from multiple vision encoders, but naively stacking them yields diminishing returns while multiplying inference costs. We propose SCOPE, a Mixture-of-Encoders (MoEnc) framework that dynamically selects one specialized encoder per image-text pair via instance-level routing, unlike token-level routing in traditional MoE. SCOPE maintains a shared encoder and a pool of routed encoders. A lightweight router uses cross-attention between text prompts and shared visual features to select the optimal encoder from the routed encoders. To train this router, we introduce dual entropy regularization with auxiliary losses to balance dataset-level load distribution with instance-level routing confidence. Remarkably, SCOPE with one shared plus one routed encoder outperforms models using all four extra encoders simultaneously, while reducing compute by 24-49\%. This demonstrates that intelligent encoder selection beats brute-force aggregation, challenging the prevailing paradigm in multi-encoder VLMs. |
| title | Scope: Selective Cross-modal Orchestration of Visual Perception Experts |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2510.12974 |