u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909298650513408 |
|---|---|
| author | Xu, Jinjin Xu, Liwu Yang, Yuzhe Li, Xiang Wang, Fanyi Xie, Yanchun Huang, Yi-Jie Li, Yaqian |
| author_facet | Xu, Jinjin Xu, Liwu Yang, Yuzhe Li, Xiang Wang, Fanyi Xie, Yanchun Huang, Yi-Jie Li, Yaqian |
| contents | Recent advancements in multi-modal large language models (MLLMs) have led to substantial improvements in visual understanding, primarily driven by sophisticated modality alignment strategies. However, predominant approaches prioritize global or regional comprehension, with less focus on fine-grained, pixel-level tasks. To address this gap, we introduce u-LLaVA, an innovative unifying multi-task framework that integrates pixel, regional, and global features to refine the perceptual faculties of MLLMs. We commence by leveraging an efficient modality alignment approach, harnessing both image and video datasets to bolster the model's foundational understanding across diverse visual contexts. Subsequently, a joint instruction tuning method with task-specific projectors and decoders for end-to-end downstream training is presented. Furthermore, this work contributes a novel mask-based multi-task dataset comprising 277K samples, crafted to challenge and assess the fine-grained perception capabilities of MLLMs. The overall framework is simple, effective, and achieves state-of-the-art performance across multiple benchmarks. We also make our model, data, and code publicly accessible at https://github.com/OPPOMKLab/u-LLaVA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_05348 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model Xu, Jinjin Xu, Liwu Yang, Yuzhe Li, Xiang Wang, Fanyi Xie, Yanchun Huang, Yi-Jie Li, Yaqian Computer Vision and Pattern Recognition Recent advancements in multi-modal large language models (MLLMs) have led to substantial improvements in visual understanding, primarily driven by sophisticated modality alignment strategies. However, predominant approaches prioritize global or regional comprehension, with less focus on fine-grained, pixel-level tasks. To address this gap, we introduce u-LLaVA, an innovative unifying multi-task framework that integrates pixel, regional, and global features to refine the perceptual faculties of MLLMs. We commence by leveraging an efficient modality alignment approach, harnessing both image and video datasets to bolster the model's foundational understanding across diverse visual contexts. Subsequently, a joint instruction tuning method with task-specific projectors and decoders for end-to-end downstream training is presented. Furthermore, this work contributes a novel mask-based multi-task dataset comprising 277K samples, crafted to challenge and assess the fine-grained perception capabilities of MLLMs. The overall framework is simple, effective, and achieves state-of-the-art performance across multiple benchmarks. We also make our model, data, and code publicly accessible at https://github.com/OPPOMKLab/u-LLaVA. |
| title | u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2311.05348 |