FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answering
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918176809287680 |
|---|---|
| author | Zhong, Liangyu Rosenthal, Fabio Sicking, Joachim Hüger, Fabian Bagdonat, Thorsten Gottschalk, Hanno Schwinn, Leo |
| author_facet | Zhong, Liangyu Rosenthal, Fabio Sicking, Joachim Hüger, Fabian Bagdonat, Thorsten Gottschalk, Hanno Schwinn, Leo |
| contents | While Multimodal Large Language Models (MLLMs) offer strong perception and reasoning capabilities for image-text input, Visual Question Answering (VQA) focusing on small image details still remains a challenge. Although visual cropping techniques seem promising, recent approaches have several limitations: the need for task-specific fine-tuning, low efficiency due to uninformed exhaustive search, or incompatibility with efficient attention implementations. We address these shortcomings by proposing a training-free visual cropping method, dubbed FOCUS, that leverages MLLM-internal representations to guide the search for the most relevant image region. This is accomplished in four steps: first, we identify the target object(s) in the VQA prompt; second, we compute an object relevance map using the key-value (KV) cache; third, we propose and rank relevant image regions based on the map; and finally, we perform the fine-grained VQA task using the top-ranked region. As a result of this informed search strategy, FOCUS achieves strong performance across four fine-grained VQA datasets and three types of MLLMs. It outperforms three popular visual cropping methods in both accuracy and efficiency, and matches the best-performing baseline, ZoomEye, while requiring 3 - 6.5 x less compute. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_21710 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answering Zhong, Liangyu Rosenthal, Fabio Sicking, Joachim Hüger, Fabian Bagdonat, Thorsten Gottschalk, Hanno Schwinn, Leo Computer Vision and Pattern Recognition While Multimodal Large Language Models (MLLMs) offer strong perception and reasoning capabilities for image-text input, Visual Question Answering (VQA) focusing on small image details still remains a challenge. Although visual cropping techniques seem promising, recent approaches have several limitations: the need for task-specific fine-tuning, low efficiency due to uninformed exhaustive search, or incompatibility with efficient attention implementations. We address these shortcomings by proposing a training-free visual cropping method, dubbed FOCUS, that leverages MLLM-internal representations to guide the search for the most relevant image region. This is accomplished in four steps: first, we identify the target object(s) in the VQA prompt; second, we compute an object relevance map using the key-value (KV) cache; third, we propose and rank relevant image regions based on the map; and finally, we perform the fine-grained VQA task using the top-ranked region. As a result of this informed search strategy, FOCUS achieves strong performance across four fine-grained VQA datasets and three types of MLLMs. It outperforms three popular visual cropping methods in both accuracy and efficiency, and matches the best-performing baseline, ZoomEye, while requiring 3 - 6.5 x less compute. |
| title | FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answering |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2506.21710 |