Decoupling Vision and Language: Codebook Anchored Visual Adaptation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908848456990720 |
|---|---|
| author | Wu, Jason Zhao, Tianchen Liu, Chang Cai, Jiarui Zhang, Zheng Li, Zhuowei Singh, Aaditya Xu, Xiang Srivastava, Mani Wu, Jonathan |
| author_facet | Wu, Jason Zhao, Tianchen Liu, Chang Cai, Jiarui Zhang, Zheng Li, Zhuowei Singh, Aaditya Xu, Xiang Srivastava, Mani Wu, Jonathan |
| contents | Large Vision-Language Models (LVLMs) use their vision encoders to translate images into representations for downstream reasoning, but the encoders often underperform in domain-specific visual tasks such as medical image diagnosis or fine-grained classification, where representation errors can cascade through the language model, leading to incorrect responses. Existing adaptation methods modify the continuous feature interface between encoder and language model through projector tuning or other parameter-efficient updates, which still couples the two components and requires re-alignment whenever the encoder changes. We introduce CRAFT (Codebook RegulAted Fine-Tuning), a lightweight method that fine-tunes the encoder using a discrete codebook that anchors visual representations to a stable token space, achieving domain adaptation without modifying other parts of the model. This decoupled design allows the adapted encoder to seamlessly boost the performance of LVLMs with different language architectures, as long as they share the same codebook. Empirically, CRAFT achieves an average gain of 13.51% across 10 domain-specific benchmarks such as VQARAD and PlantVillage, while preserving the LLM's linguistic capabilities and outperforming peer methods that operate on continuous tokens. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_19449 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Decoupling Vision and Language: Codebook Anchored Visual Adaptation Wu, Jason Zhao, Tianchen Liu, Chang Cai, Jiarui Zhang, Zheng Li, Zhuowei Singh, Aaditya Xu, Xiang Srivastava, Mani Wu, Jonathan Computer Vision and Pattern Recognition I.2.10; I.4.8; I.2.6; I.5.1 Large Vision-Language Models (LVLMs) use their vision encoders to translate images into representations for downstream reasoning, but the encoders often underperform in domain-specific visual tasks such as medical image diagnosis or fine-grained classification, where representation errors can cascade through the language model, leading to incorrect responses. Existing adaptation methods modify the continuous feature interface between encoder and language model through projector tuning or other parameter-efficient updates, which still couples the two components and requires re-alignment whenever the encoder changes. We introduce CRAFT (Codebook RegulAted Fine-Tuning), a lightweight method that fine-tunes the encoder using a discrete codebook that anchors visual representations to a stable token space, achieving domain adaptation without modifying other parts of the model. This decoupled design allows the adapted encoder to seamlessly boost the performance of LVLMs with different language architectures, as long as they share the same codebook. Empirically, CRAFT achieves an average gain of 13.51% across 10 domain-specific benchmarks such as VQARAD and PlantVillage, while preserving the LLM's linguistic capabilities and outperforming peer methods that operate on continuous tokens. |
| title | Decoupling Vision and Language: Codebook Anchored Visual Adaptation |
| topic | Computer Vision and Pattern Recognition I.2.10; I.4.8; I.2.6; I.5.1 |
| url | https://arxiv.org/abs/2602.19449 |