LLaVA-Read: Enhancing Reading Ability of Multimodal Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914889862217728 |
|---|---|
| author | Zhang, Ruiyi Zhou, Yufan Chen, Jian Gu, Jiuxiang Chen, Changyou Sun, Tong |
| author_facet | Zhang, Ruiyi Zhou, Yufan Chen, Jian Gu, Jiuxiang Chen, Changyou Sun, Tong |
| contents | Large multimodal language models have demonstrated impressive capabilities in understanding and manipulating images. However, many of these models struggle with comprehending intensive textual contents embedded within the images, primarily due to the limited text recognition and layout understanding ability. To understand the sources of these limitations, we perform an exploratory analysis showing the drawbacks of classical visual encoders on visual text understanding. Hence, we present LLaVA-Read, a multimodal large language model that utilizes dual visual encoders along with a visual text encoder. Our model surpasses existing state-of-the-art models in various text-rich image understanding tasks, showcasing enhanced comprehension of textual content within images. Together, our research suggests visual text understanding remains an open challenge and an efficient visual text encoder is crucial for future successful multimodal systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_19185 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLaVA-Read: Enhancing Reading Ability of Multimodal Language Models Zhang, Ruiyi Zhou, Yufan Chen, Jian Gu, Jiuxiang Chen, Changyou Sun, Tong Computer Vision and Pattern Recognition Artificial Intelligence Large multimodal language models have demonstrated impressive capabilities in understanding and manipulating images. However, many of these models struggle with comprehending intensive textual contents embedded within the images, primarily due to the limited text recognition and layout understanding ability. To understand the sources of these limitations, we perform an exploratory analysis showing the drawbacks of classical visual encoders on visual text understanding. Hence, we present LLaVA-Read, a multimodal large language model that utilizes dual visual encoders along with a visual text encoder. Our model surpasses existing state-of-the-art models in various text-rich image understanding tasks, showcasing enhanced comprehension of textual content within images. Together, our research suggests visual text understanding remains an open challenge and an efficient visual text encoder is crucial for future successful multimodal systems. |
| title | LLaVA-Read: Enhancing Reading Ability of Multimodal Language Models |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2407.19185 |