Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909626130235392 |
|---|---|
| author | Li, Xudong Zhang, Mengdan Chen, Peixian Zheng, Xiawu Zhang, Yan Zheng, Jingyuan Shen, Yunhang Li, Ke Fu, Chaoyou Sun, Xing Ji, Rongrong |
| author_facet | Li, Xudong Zhang, Mengdan Chen, Peixian Zheng, Xiawu Zhang, Yan Zheng, Jingyuan Shen, Yunhang Li, Ke Fu, Chaoyou Sun, Xing Ji, Rongrong |
| contents | Multi-modal Large Language Models (MLLMs) excel at single-image tasks but struggle with multi-image understanding due to cross-modal misalignment, leading to hallucinations (context omission, conflation, and misinterpretation). Existing methods using Direct Preference Optimization (DPO) constrain optimization to a solitary image reference within the input sequence, neglecting holistic context modeling. We propose Context-to-Cue Direct Preference Optimization (CcDPO), a multi-level preference optimization framework that enhances per-image perception in multi-image settings by zooming into visual clues -- from sequential context to local details. It features: (i) Context-Level Optimization : Re-evaluates cognitive biases underlying MLLMs' multi-image context comprehension and integrates a spectrum of low-cost global sequence preferences for bias mitigation. (ii) Needle-Level Optimization : Directs attention to fine-grained visual details through region-targeted visual prompts and multimodal preference supervision. To support scalable optimization, we also construct MultiScope-42k, an automatically generated dataset with high-quality multi-level preference pairs. Experiments show that CcDPO significantly reduces hallucinations and yields consistent performance gains across general single- and multi-image tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_22396 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs Li, Xudong Zhang, Mengdan Chen, Peixian Zheng, Xiawu Zhang, Yan Zheng, Jingyuan Shen, Yunhang Li, Ke Fu, Chaoyou Sun, Xing Ji, Rongrong Computer Vision and Pattern Recognition Multi-modal Large Language Models (MLLMs) excel at single-image tasks but struggle with multi-image understanding due to cross-modal misalignment, leading to hallucinations (context omission, conflation, and misinterpretation). Existing methods using Direct Preference Optimization (DPO) constrain optimization to a solitary image reference within the input sequence, neglecting holistic context modeling. We propose Context-to-Cue Direct Preference Optimization (CcDPO), a multi-level preference optimization framework that enhances per-image perception in multi-image settings by zooming into visual clues -- from sequential context to local details. It features: (i) Context-Level Optimization : Re-evaluates cognitive biases underlying MLLMs' multi-image context comprehension and integrates a spectrum of low-cost global sequence preferences for bias mitigation. (ii) Needle-Level Optimization : Directs attention to fine-grained visual details through region-targeted visual prompts and multimodal preference supervision. To support scalable optimization, we also construct MultiScope-42k, an automatically generated dataset with high-quality multi-level preference pairs. Experiments show that CcDPO significantly reduces hallucinations and yields consistent performance gains across general single- and multi-image tasks. |
| title | Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.22396 |