Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.20906 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917762574581760 |
|---|---|
| author | TG, Adithya SK, Adithya Bharadwaj, Abhinav R HA, Abhiram Narayan, Surabhi |
| author_facet | TG, Adithya SK, Adithya Bharadwaj, Abhinav R HA, Abhiram Narayan, Surabhi |
| contents | Interacting and understanding with text heavy visual content with multiple images is a major challenge for traditional vision models. This paper is on enhancing vision models' capability to comprehend or understand and learn from images containing a huge amount of textual information from the likes of textbooks and research papers which contain multiple images like graphs, etc and tables in them with different types of axes and scales. The approach involves dataset preprocessing, fine tuning which is by using instructional oriented data and evaluation. We also built a visual chat application integrating CLIP for image encoding and a model from the Massive Text Embedding Benchmark which is developed to consider both textual and visual inputs. An accuracy of 96.71% was obtained. The aim of the project is to increase and also enhance the advance vision models' capabilities in understanding complex visual textual data interconnected data, contributing to multimodal AI. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_20906 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhancing Vision Models for Text-Heavy Content Understanding and Interaction TG, Adithya SK, Adithya Bharadwaj, Abhinav R HA, Abhiram Narayan, Surabhi Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Interacting and understanding with text heavy visual content with multiple images is a major challenge for traditional vision models. This paper is on enhancing vision models' capability to comprehend or understand and learn from images containing a huge amount of textual information from the likes of textbooks and research papers which contain multiple images like graphs, etc and tables in them with different types of axes and scales. The approach involves dataset preprocessing, fine tuning which is by using instructional oriented data and evaluation. We also built a visual chat application integrating CLIP for image encoding and a model from the Massive Text Embedding Benchmark which is developed to consider both textual and visual inputs. An accuracy of 96.71% was obtained. The aim of the project is to increase and also enhance the advance vision models' capabilities in understanding complex visual textual data interconnected data, contributing to multimodal AI. |
| title | Enhancing Vision Models for Text-Heavy Content Understanding and Interaction |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2405.20906 |