Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913918269521920 |
|---|---|
| author | Yan, Ziang Li, Zhilin He, Yinan Wang, Chenting Li, Kunchang Li, Xinhao Zeng, Xiangyu Wang, Zilei Wang, Yali Qiao, Yu Wang, Limin Wang, Yi |
| author_facet | Yan, Ziang Li, Zhilin He, Yinan Wang, Chenting Li, Kunchang Li, Xinhao Zeng, Xiangyu Wang, Zilei Wang, Yali Qiao, Yu Wang, Limin Wang, Yi |
| contents | Current multimodal large language models (MLLMs) struggle with fine-grained or precise understanding of visuals although they give comprehensive perception and reasoning in a spectrum of vision applications. Recent studies either develop tool-using or unify specific visual tasks into the autoregressive framework, often at the expense of overall multimodal performance. To address this issue and enhance MLLMs with visual tasks in a scalable fashion, we propose Task Preference Optimization (TPO), a novel method that utilizes differentiable task preferences derived from typical fine-grained visual tasks. TPO introduces learnable task tokens that establish connections between multiple task-specific heads and the MLLM. By leveraging rich visual labels during training, TPO significantly enhances the MLLM's multimodal capabilities and task-specific performance. Through multi-task co-training within TPO, we observe synergistic benefits that elevate individual task performance beyond what is achievable through single-task training methodologies. Our instantiation of this approach with VideoChat and LLaVA demonstrates an overall 14.6% improvement in multimodal performance compared to baseline models. Additionally, MLLM-TPO demonstrates robust zero-shot capabilities across various tasks, performing comparably to state-of-the-art supervised models. The code will be released at https://github.com/OpenGVLab/TPO |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_19326 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment Yan, Ziang Li, Zhilin He, Yinan Wang, Chenting Li, Kunchang Li, Xinhao Zeng, Xiangyu Wang, Zilei Wang, Yali Qiao, Yu Wang, Limin Wang, Yi Computer Vision and Pattern Recognition Current multimodal large language models (MLLMs) struggle with fine-grained or precise understanding of visuals although they give comprehensive perception and reasoning in a spectrum of vision applications. Recent studies either develop tool-using or unify specific visual tasks into the autoregressive framework, often at the expense of overall multimodal performance. To address this issue and enhance MLLMs with visual tasks in a scalable fashion, we propose Task Preference Optimization (TPO), a novel method that utilizes differentiable task preferences derived from typical fine-grained visual tasks. TPO introduces learnable task tokens that establish connections between multiple task-specific heads and the MLLM. By leveraging rich visual labels during training, TPO significantly enhances the MLLM's multimodal capabilities and task-specific performance. Through multi-task co-training within TPO, we observe synergistic benefits that elevate individual task performance beyond what is achievable through single-task training methodologies. Our instantiation of this approach with VideoChat and LLaVA demonstrates an overall 14.6% improvement in multimodal performance compared to baseline models. Additionally, MLLM-TPO demonstrates robust zero-shot capabilities across various tasks, performing comparably to state-of-the-art supervised models. The code will be released at https://github.com/OpenGVLab/TPO |
| title | Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.19326 |