M2IST: Multi-Modal Interactive Side-Tuning for Efficient Referring Expression Comprehension
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913734005358592 |
|---|---|
| author | Liu, Xuyang Liu, Ting Huang, Siteng Xin, Yi Hu, Yue Yin, Quanjun Wang, Donglin Wu, Yuanyuan Chen, Honggang |
| author_facet | Liu, Xuyang Liu, Ting Huang, Siteng Xin, Yi Hu, Yue Yin, Quanjun Wang, Donglin Wu, Yuanyuan Chen, Honggang |
| contents | Referring expression comprehension (REC) is a vision-language task to locate a target object in an image based on a language expression. Fully fine-tuning general-purpose pre-trained vision-language foundation models for REC yields impressive performance but becomes increasingly costly. Parameter-efficient transfer learning (PETL) methods have shown strong performance with fewer tunable parameters. However, directly applying PETL to REC faces two challenges: (1) insufficient multi-modal interaction between pre-trained vision-language foundation models, and (2) high GPU memory usage due to gradients passing through the heavy vision-language foundation models. To this end, we present M2IST: Multi-Modal Interactive Side-Tuning with M3ISAs: Mixture of Multi-Modal Interactive Side-Adapters. During fine-tuning, we fix the pre-trained uni-modal encoders and update M3ISAs to enable efficient vision-language alignment for REC. Empirical results reveal that M2IST achieves better performance-efficiency trade-off than full fine-tuning and other PETL methods, requiring only 2.11\% tunable parameters, 39.61\% GPU memory, and 63.46\% training time while maintaining competitive performance. Our code is released at https://github.com/xuyang-liu16/M2IST. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_01131 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | M2IST: Multi-Modal Interactive Side-Tuning for Efficient Referring Expression Comprehension Liu, Xuyang Liu, Ting Huang, Siteng Xin, Yi Hu, Yue Yin, Quanjun Wang, Donglin Wu, Yuanyuan Chen, Honggang Computer Vision and Pattern Recognition Referring expression comprehension (REC) is a vision-language task to locate a target object in an image based on a language expression. Fully fine-tuning general-purpose pre-trained vision-language foundation models for REC yields impressive performance but becomes increasingly costly. Parameter-efficient transfer learning (PETL) methods have shown strong performance with fewer tunable parameters. However, directly applying PETL to REC faces two challenges: (1) insufficient multi-modal interaction between pre-trained vision-language foundation models, and (2) high GPU memory usage due to gradients passing through the heavy vision-language foundation models. To this end, we present M2IST: Multi-Modal Interactive Side-Tuning with M3ISAs: Mixture of Multi-Modal Interactive Side-Adapters. During fine-tuning, we fix the pre-trained uni-modal encoders and update M3ISAs to enable efficient vision-language alignment for REC. Empirical results reveal that M2IST achieves better performance-efficiency trade-off than full fine-tuning and other PETL methods, requiring only 2.11\% tunable parameters, 39.61\% GPU memory, and 63.46\% training time while maintaining competitive performance. Our code is released at https://github.com/xuyang-liu16/M2IST. |
| title | M2IST: Multi-Modal Interactive Side-Tuning for Efficient Referring Expression Comprehension |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.01131 |