HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911805395173376 |
|---|---|
| author | Zhang, Wenqiao Lin, Tianwei Liu, Jiang Shu, Fangxun Li, Haoyuan Zhang, Lei Wanggui, He Zhou, Hao Lv, Zheqi Jiang, Hao Li, Juncheng Tang, Siliang Zhuang, Yueting |
| author_facet | Zhang, Wenqiao Lin, Tianwei Liu, Jiang Shu, Fangxun Li, Haoyuan Zhang, Lei Wanggui, He Zhou, Hao Lv, Zheqi Jiang, Hao Li, Juncheng Tang, Siliang Zhuang, Yueting |
| contents | Recent advancements indicate that scaling up Multimodal Large Language Models (MLLMs) effectively enhances performance on downstream multimodal tasks. The prevailing MLLM paradigm, \emph{e.g.}, LLaVA, transforms visual features into text-like tokens using a \emph{static} vision-language mapper, thereby enabling \emph{static} LLMs to develop the capability to comprehend visual information through visual instruction tuning. Although promising, the \emph{static} tuning strategy~\footnote{The static tuning refers to the trained model with static parameters.} that shares the same parameters may constrain performance across different downstream multimodal tasks. In light of this, we introduce HyperLLaVA, which involves adaptive tuning of the projector and LLM parameters, in conjunction with a dynamic visual expert and language expert, respectively. These experts are derived from HyperNetworks, which generates adaptive parameter shifts through visual and language guidance, enabling dynamic projector and LLM modeling in two-stage training.
Our experiments demonstrate that our solution significantly surpasses LLaVA on existing MLLM benchmarks, including MME, MMBench, SEED-Bench, and LLaVA-Bench. ~\footnote{Our project is available on the link https://github.com/DCDmllm/HyperLLaVA}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_13447 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal Large Language Models Zhang, Wenqiao Lin, Tianwei Liu, Jiang Shu, Fangxun Li, Haoyuan Zhang, Lei Wanggui, He Zhou, Hao Lv, Zheqi Jiang, Hao Li, Juncheng Tang, Siliang Zhuang, Yueting Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Recent advancements indicate that scaling up Multimodal Large Language Models (MLLMs) effectively enhances performance on downstream multimodal tasks. The prevailing MLLM paradigm, \emph{e.g.}, LLaVA, transforms visual features into text-like tokens using a \emph{static} vision-language mapper, thereby enabling \emph{static} LLMs to develop the capability to comprehend visual information through visual instruction tuning. Although promising, the \emph{static} tuning strategy~\footnote{The static tuning refers to the trained model with static parameters.} that shares the same parameters may constrain performance across different downstream multimodal tasks. In light of this, we introduce HyperLLaVA, which involves adaptive tuning of the projector and LLM parameters, in conjunction with a dynamic visual expert and language expert, respectively. These experts are derived from HyperNetworks, which generates adaptive parameter shifts through visual and language guidance, enabling dynamic projector and LLM modeling in two-stage training. Our experiments demonstrate that our solution significantly surpasses LLaVA on existing MLLM benchmarks, including MME, MMBench, SEED-Bench, and LLaVA-Bench. ~\footnote{Our project is available on the link https://github.com/DCDmllm/HyperLLaVA}. |
| title | HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal Large Language Models |
| topic | Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.13447 |