Pixel-Grounded Retrieval for Knowledgeable Large Multimodal Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917224948695040 |
|---|---|
| author | Kim, Jeonghwan Tao, Renjie Sharma, Sanat Wang, Jiaqi Sun, Kai Lin, Zhaojiang Moon, Seungwhan Mathias, Lambert Kumar, Anuj Ji, Heng Dong, Xin Luna |
| author_facet | Kim, Jeonghwan Tao, Renjie Sharma, Sanat Wang, Jiaqi Sun, Kai Lin, Zhaojiang Moon, Seungwhan Mathias, Lambert Kumar, Anuj Ji, Heng Dong, Xin Luna |
| contents | Visual Question Answering (VQA) often requires coupling fine-grained perception with factual knowledge beyond the input image. Prior multimodal Retrieval-Augmented Generation (MM-RAG) systems improve factual grounding but lack an internal policy for when and how to retrieve. We propose PixSearch, the first end-to-end Segmenting Large Multimodal Model (LMM) that unifies region-level perception and retrieval-augmented reasoning. During encoding, PixSearch emits <search> tokens to trigger retrieval, selects query modalities (text, image, or region), and generates pixel-level masks that directly serve as visual queries, eliminating the reliance on modular pipelines (detectors, segmenters, captioners, etc.). A two-stage supervised fine-tuning regimen with search-interleaved supervision teaches retrieval timing and query selection while preserving segmentation ability. On egocentric and entity-centric VQA benchmarks, PixSearch substantially improves factual consistency and generalization, yielding a 19.7% relative gain in accuracy on CRAG-MM compared to whole image retrieval, while retaining competitive reasoning performance on various VQA and text-only QA tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_19060 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Pixel-Grounded Retrieval for Knowledgeable Large Multimodal Models Kim, Jeonghwan Tao, Renjie Sharma, Sanat Wang, Jiaqi Sun, Kai Lin, Zhaojiang Moon, Seungwhan Mathias, Lambert Kumar, Anuj Ji, Heng Dong, Xin Luna Computer Vision and Pattern Recognition Artificial Intelligence Visual Question Answering (VQA) often requires coupling fine-grained perception with factual knowledge beyond the input image. Prior multimodal Retrieval-Augmented Generation (MM-RAG) systems improve factual grounding but lack an internal policy for when and how to retrieve. We propose PixSearch, the first end-to-end Segmenting Large Multimodal Model (LMM) that unifies region-level perception and retrieval-augmented reasoning. During encoding, PixSearch emits <search> tokens to trigger retrieval, selects query modalities (text, image, or region), and generates pixel-level masks that directly serve as visual queries, eliminating the reliance on modular pipelines (detectors, segmenters, captioners, etc.). A two-stage supervised fine-tuning regimen with search-interleaved supervision teaches retrieval timing and query selection while preserving segmentation ability. On egocentric and entity-centric VQA benchmarks, PixSearch substantially improves factual consistency and generalization, yielding a 19.7% relative gain in accuracy on CRAG-MM compared to whole image retrieval, while retaining competitive reasoning performance on various VQA and text-only QA tasks. |
| title | Pixel-Grounded Retrieval for Knowledgeable Large Multimodal Models |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2601.19060 |