OmniFusion Technical Report
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913307318812672 |
|---|---|
| author | Goncharova, Elizaveta Razzhigaev, Anton Mikhalchuk, Matvey Kurkin, Maxim Abdullaeva, Irina Skripkin, Matvey Oseledets, Ivan Dimitrov, Denis Kuznetsov, Andrey |
| author_facet | Goncharova, Elizaveta Razzhigaev, Anton Mikhalchuk, Matvey Kurkin, Maxim Abdullaeva, Irina Skripkin, Matvey Oseledets, Ivan Dimitrov, Denis Kuznetsov, Andrey |
| contents | Last year, multimodal architectures served up a revolution in AI-based approaches and solutions, extending the capabilities of large language models (LLM). We propose an \textit{OmniFusion} model based on a pretrained LLM and adapters for visual modality. We evaluated and compared several architecture design principles for better text and visual data coupling: MLP and transformer adapters, various CLIP ViT-based encoders (SigLIP, InternVIT, etc.), and their fusing approach, image encoding method (whole image or tiles encoding) and two 7B LLMs (the proprietary one and open-source Mistral). Experiments on 8 visual-language benchmarks show the top score for the best OmniFusion setup in terms of different VQA tasks in comparison with open-source LLaVA-like solutions: VizWiz, Pope, MM-Vet, ScienceQA, MMBench, TextVQA, VQAv2, MMMU. We also propose a variety of situations, where OmniFusion provides highly-detailed answers in different domains: housekeeping, sightseeing, culture, medicine, handwritten and scanned equations recognition, etc. Mistral-based OmniFusion model is an open-source solution with weights, training and inference scripts available at https://github.com/AIRI-Institute/OmniFusion. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_06212 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | OmniFusion Technical Report Goncharova, Elizaveta Razzhigaev, Anton Mikhalchuk, Matvey Kurkin, Maxim Abdullaeva, Irina Skripkin, Matvey Oseledets, Ivan Dimitrov, Denis Kuznetsov, Andrey Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning 6804, 68T50 (Primary) I.2.7; I.2.10; I.4.9 Last year, multimodal architectures served up a revolution in AI-based approaches and solutions, extending the capabilities of large language models (LLM). We propose an \textit{OmniFusion} model based on a pretrained LLM and adapters for visual modality. We evaluated and compared several architecture design principles for better text and visual data coupling: MLP and transformer adapters, various CLIP ViT-based encoders (SigLIP, InternVIT, etc.), and their fusing approach, image encoding method (whole image or tiles encoding) and two 7B LLMs (the proprietary one and open-source Mistral). Experiments on 8 visual-language benchmarks show the top score for the best OmniFusion setup in terms of different VQA tasks in comparison with open-source LLaVA-like solutions: VizWiz, Pope, MM-Vet, ScienceQA, MMBench, TextVQA, VQAv2, MMMU. We also propose a variety of situations, where OmniFusion provides highly-detailed answers in different domains: housekeeping, sightseeing, culture, medicine, handwritten and scanned equations recognition, etc. Mistral-based OmniFusion model is an open-source solution with weights, training and inference scripts available at https://github.com/AIRI-Institute/OmniFusion. |
| title | OmniFusion Technical Report |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning 6804, 68T50 (Primary) I.2.7; I.2.10; I.4.9 |
| url | https://arxiv.org/abs/2404.06212 |