Re-Imagining Multimodal Instruction Tuning: A Representation View
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912285299048448 |
|---|---|
| author | Liu, Yiyang Liang, James Chenhao Tang, Ruixiang Lee, Yugyung Rabbani, Majid Dianat, Sohail Rao, Raghuveer Huang, Lifu Liu, Dongfang Wang, Qifan Han, Cheng |
| author_facet | Liu, Yiyang Liang, James Chenhao Tang, Ruixiang Lee, Yugyung Rabbani, Majid Dianat, Sohail Rao, Raghuveer Huang, Lifu Liu, Dongfang Wang, Qifan Han, Cheng |
| contents | Multimodal instruction tuning has proven to be an effective strategy for achieving zero-shot generalization by fine-tuning pre-trained Large Multimodal Models (LMMs) with instruction-following data. However, as the scale of LMMs continues to grow, fully fine-tuning these models has become highly parameter-intensive. Although Parameter-Efficient Fine-Tuning (PEFT) methods have been introduced to reduce the number of tunable parameters, a significant performance gap remains compared to full fine-tuning. Furthermore, existing PEFT approaches are often highly parameterized, making them difficult to interpret and control. In light of this, we introduce Multimodal Representation Tuning (MRT), a novel approach that focuses on directly editing semantically rich multimodal representations to achieve strong performance and provide intuitive control over LMMs. Empirical results show that our method surpasses current state-of-the-art baselines with significant performance gains (e.g., 1580.40 MME score) while requiring substantially fewer tunable parameters (e.g., 0.03% parameters). Additionally, we conduct experiments on editing instrumental tokens within multimodal representations, demonstrating that direct manipulation of these representations enables simple yet effective control over network behavior. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_00723 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Re-Imagining Multimodal Instruction Tuning: A Representation View Liu, Yiyang Liang, James Chenhao Tang, Ruixiang Lee, Yugyung Rabbani, Majid Dianat, Sohail Rao, Raghuveer Huang, Lifu Liu, Dongfang Wang, Qifan Han, Cheng Machine Learning Multimodal instruction tuning has proven to be an effective strategy for achieving zero-shot generalization by fine-tuning pre-trained Large Multimodal Models (LMMs) with instruction-following data. However, as the scale of LMMs continues to grow, fully fine-tuning these models has become highly parameter-intensive. Although Parameter-Efficient Fine-Tuning (PEFT) methods have been introduced to reduce the number of tunable parameters, a significant performance gap remains compared to full fine-tuning. Furthermore, existing PEFT approaches are often highly parameterized, making them difficult to interpret and control. In light of this, we introduce Multimodal Representation Tuning (MRT), a novel approach that focuses on directly editing semantically rich multimodal representations to achieve strong performance and provide intuitive control over LMMs. Empirical results show that our method surpasses current state-of-the-art baselines with significant performance gains (e.g., 1580.40 MME score) while requiring substantially fewer tunable parameters (e.g., 0.03% parameters). Additionally, we conduct experiments on editing instrumental tokens within multimodal representations, demonstrating that direct manipulation of these representations enables simple yet effective control over network behavior. |
| title | Re-Imagining Multimodal Instruction Tuning: A Representation View |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2503.00723 |