Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.26937 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912678520291328 |
|---|---|
| author | Huang, Zimeng Ke, Jinxin Fan, Xiaoxuan Yang, Yufeng Liu, Yang Zhonghan, Liu Wang, Zedi Dai, Junteng Jiang, Haoyi Zhou, Yuyu Wang, Keze Chen, Ziliang |
| author_facet | Huang, Zimeng Ke, Jinxin Fan, Xiaoxuan Yang, Yufeng Liu, Yang Zhonghan, Liu Wang, Zedi Dai, Junteng Jiang, Haoyi Zhou, Yuyu Wang, Keze Chen, Ziliang |
| contents | Large Vision-Language Models (LVLMs) have exhibited remarkable progress. However, deficiencies remain compared to human intelligence, such as hallucination and shallow pattern matching. In this work, we aim to evaluate a fundamental yet underexplored intelligence: association, a cornerstone of human cognition for creative thinking and knowledge integration. Current benchmarks, often limited to closed-ended tasks, fail to capture the complexity of open-ended association reasoning vital for real-world applications. To address this, we present MM-OPERA, a systematic benchmark with 11,497 instances across two open-ended tasks: Remote-Item Association (RIA) and In-Context Association (ICA), aligning association intelligence evaluation with human psychometric principles. It challenges LVLMs to resemble the spirit of divergent thinking and convergent associative reasoning through free-form responses and explicit reasoning paths. We deploy tailored LLM-as-a-Judge strategies to evaluate open-ended outputs, applying process-reward-informed judgment to dissect reasoning with precision. Extensive empirical studies on state-of-the-art LVLMs, including sensitivity analysis of task instances, validity analysis of LLM-as-a-Judge strategies, and diversity analysis across abilities, domains, languages, cultures, etc., provide a comprehensive and nuanced understanding of the limitations of current LVLMs in associative reasoning, paving the way for more human-like and general-purpose AI. The dataset and code are available at https://github.com/MM-OPERA-Bench/MM-OPERA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_26937 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MM-OPERA: Benchmarking Open-ended Association Reasoning for Large Vision-Language Models Huang, Zimeng Ke, Jinxin Fan, Xiaoxuan Yang, Yufeng Liu, Yang Zhonghan, Liu Wang, Zedi Dai, Junteng Jiang, Haoyi Zhou, Yuyu Wang, Keze Chen, Ziliang Machine Learning Large Vision-Language Models (LVLMs) have exhibited remarkable progress. However, deficiencies remain compared to human intelligence, such as hallucination and shallow pattern matching. In this work, we aim to evaluate a fundamental yet underexplored intelligence: association, a cornerstone of human cognition for creative thinking and knowledge integration. Current benchmarks, often limited to closed-ended tasks, fail to capture the complexity of open-ended association reasoning vital for real-world applications. To address this, we present MM-OPERA, a systematic benchmark with 11,497 instances across two open-ended tasks: Remote-Item Association (RIA) and In-Context Association (ICA), aligning association intelligence evaluation with human psychometric principles. It challenges LVLMs to resemble the spirit of divergent thinking and convergent associative reasoning through free-form responses and explicit reasoning paths. We deploy tailored LLM-as-a-Judge strategies to evaluate open-ended outputs, applying process-reward-informed judgment to dissect reasoning with precision. Extensive empirical studies on state-of-the-art LVLMs, including sensitivity analysis of task instances, validity analysis of LLM-as-a-Judge strategies, and diversity analysis across abilities, domains, languages, cultures, etc., provide a comprehensive and nuanced understanding of the limitations of current LVLMs in associative reasoning, paving the way for more human-like and general-purpose AI. The dataset and code are available at https://github.com/MM-OPERA-Bench/MM-OPERA. |
| title | MM-OPERA: Benchmarking Open-ended Association Reasoning for Large Vision-Language Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2510.26937 |