Law of Vision Representation in MLLMs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908576039043072 |
|---|---|
| author | Yang, Shijia Zhai, Bohan You, Quanzeng Yuan, Jianbo Yang, Hongxia Xu, Chenfeng |
| author_facet | Yang, Shijia Zhai, Bohan You, Quanzeng Yuan, Jianbo Yang, Hongxia Xu, Chenfeng |
| contents | We present the "Law of Vision Representation" in multimodal large language models (MLLMs). It reveals a strong correlation between the combination of cross-modal alignment, correspondence in vision representation, and MLLM performance. We quantify the two factors using the cross-modal Alignment and Correspondence score (AC score). Through extensive experiments involving thirteen different vision representation settings and evaluations across eight benchmarks, we find that the AC score is linearly correlated to model performance. By leveraging this relationship, we are able to identify and train the optimal vision representation only, which does not require finetuning the language model every time, resulting in a 99.7% reduction in computational cost. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_16357 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Law of Vision Representation in MLLMs Yang, Shijia Zhai, Bohan You, Quanzeng Yuan, Jianbo Yang, Hongxia Xu, Chenfeng Computer Vision and Pattern Recognition We present the "Law of Vision Representation" in multimodal large language models (MLLMs). It reveals a strong correlation between the combination of cross-modal alignment, correspondence in vision representation, and MLLM performance. We quantify the two factors using the cross-modal Alignment and Correspondence score (AC score). Through extensive experiments involving thirteen different vision representation settings and evaluations across eight benchmarks, we find that the AC score is linearly correlated to model performance. By leveraging this relationship, we are able to identify and train the optimal vision representation only, which does not require finetuning the language model every time, resulting in a 99.7% reduction in computational cost. |
| title | Law of Vision Representation in MLLMs |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.16357 |