Describe Anything Model for Visual Question Answering on Text-rich Images
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912515551657984 |
|---|---|
| author | Vu, Yen-Linh Duong, Dinh-Thang Duong, Truong-Binh Nguyen, Anh-Khoi Nguyen, Thanh-Huy Nguyen, Le Thien Phuc Xing, Jianhua Li, Xingjian Wang, Tianyang Bagci, Ulas Xu, Min |
| author_facet | Vu, Yen-Linh Duong, Dinh-Thang Duong, Truong-Binh Nguyen, Anh-Khoi Nguyen, Thanh-Huy Nguyen, Le Thien Phuc Xing, Jianhua Li, Xingjian Wang, Tianyang Bagci, Ulas Xu, Min |
| contents | Recent progress has been made in region-aware vision-language modeling, particularly with the emergence of the Describe Anything Model (DAM). DAM is capable of generating detailed descriptions of any specific image areas or objects without the need for additional localized image-text alignment supervision. We hypothesize that such region-level descriptive capability is beneficial for the task of Visual Question Answering (VQA), especially in challenging scenarios involving images with dense text. In such settings, the fine-grained extraction of textual information is crucial to producing correct answers. Motivated by this, we introduce DAM-QA, a framework with a tailored evaluation protocol, developed to investigate and harness the region-aware capabilities from DAM for the text-rich VQA problem that requires reasoning over text-based information within images. DAM-QA incorporates a mechanism that aggregates answers from multiple regional views of image content, enabling more effective identification of evidence that may be tied to text-related elements. Experiments on six VQA benchmarks show that our approach consistently outperforms the baseline DAM, with a notable 7+ point gain on DocVQA. DAM-QA also achieves the best overall performance among region-aware models with fewer parameters, significantly narrowing the gap with strong generalist VLMs. These results highlight the potential of DAM-like models for text-rich and broader VQA tasks when paired with efficient usage and integration strategies. Our code is publicly available at https://github.com/Linvyl/DAM-QA.git. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_12441 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Describe Anything Model for Visual Question Answering on Text-rich Images Vu, Yen-Linh Duong, Dinh-Thang Duong, Truong-Binh Nguyen, Anh-Khoi Nguyen, Thanh-Huy Nguyen, Le Thien Phuc Xing, Jianhua Li, Xingjian Wang, Tianyang Bagci, Ulas Xu, Min Computer Vision and Pattern Recognition Machine Learning Recent progress has been made in region-aware vision-language modeling, particularly with the emergence of the Describe Anything Model (DAM). DAM is capable of generating detailed descriptions of any specific image areas or objects without the need for additional localized image-text alignment supervision. We hypothesize that such region-level descriptive capability is beneficial for the task of Visual Question Answering (VQA), especially in challenging scenarios involving images with dense text. In such settings, the fine-grained extraction of textual information is crucial to producing correct answers. Motivated by this, we introduce DAM-QA, a framework with a tailored evaluation protocol, developed to investigate and harness the region-aware capabilities from DAM for the text-rich VQA problem that requires reasoning over text-based information within images. DAM-QA incorporates a mechanism that aggregates answers from multiple regional views of image content, enabling more effective identification of evidence that may be tied to text-related elements. Experiments on six VQA benchmarks show that our approach consistently outperforms the baseline DAM, with a notable 7+ point gain on DocVQA. DAM-QA also achieves the best overall performance among region-aware models with fewer parameters, significantly narrowing the gap with strong generalist VLMs. These results highlight the potential of DAM-like models for text-rich and broader VQA tasks when paired with efficient usage and integration strategies. Our code is publicly available at https://github.com/Linvyl/DAM-QA.git. |
| title | Describe Anything Model for Visual Question Answering on Text-rich Images |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2507.12441 |