Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.21662 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912964127227904 |
|---|---|
| author | Xiong, Tianyi Ge, Yi Li, Ming Zhang, Zuolong Kulkarni, Pranav Wang, Kaishen He, Qi Zhu, Zeying Liu, Chenxi Chen, Ruibo Zheng, Tong Chen, Yanshuo Wang, Xiyao Zhang, Renrui Chen, Wenhu Huang, Heng |
| author_facet | Xiong, Tianyi Ge, Yi Li, Ming Zhang, Zuolong Kulkarni, Pranav Wang, Kaishen He, Qi Zhu, Zeying Liu, Chenxi Chen, Ruibo Zheng, Tong Chen, Yanshuo Wang, Xiyao Zhang, Renrui Chen, Wenhu Huang, Heng |
| contents | Large multimodal models (LMMs) are increasingly adopted as judges in multimodal evaluation systems due to their strong instruction following and consistency with human preferences. However, their ability to follow diverse, fine-grained evaluation criteria remains underexplored. We develop Multi-Crit, a benchmark for evaluating multimodal judges on their capacity to follow pluralistic criteria and produce reliable criterion-level judgments. Covering both open-ended generation and verifiable reasoning tasks, Multi-Crit is built through a rigorous data curation pipeline that gathers challenging response pairs with multi-criterion human annotations. It further introduces three novel metrics for systematically assessing pluralistic adherence, criterion-switching flexibility, and the ability to recognize criterion-level preference conflicts. Comprehensive analysis of 25 LMMs reveals that 1) proprietary models still struggle to maintain consistent adherence to pluralistic criteria--especially in open-ended evaluation; 2) open-source models lag further behind in flexibly following diverse criteria; and 3) critic fine-tuning with holistic judgment signals enhances visual grounding but fails to generalize to pluralistic criterion-level judgment. Additional analyses on reasoning fine-tuning, test-time scaling, and boundary consistency between open-source and proprietary models further probe the limits of current multimodal judges. As a pioneering study, Multi-Crit lays the foundation for building reliable and steerable multimodal AI evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_21662 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Following Xiong, Tianyi Ge, Yi Li, Ming Zhang, Zuolong Kulkarni, Pranav Wang, Kaishen He, Qi Zhu, Zeying Liu, Chenxi Chen, Ruibo Zheng, Tong Chen, Yanshuo Wang, Xiyao Zhang, Renrui Chen, Wenhu Huang, Heng Computer Vision and Pattern Recognition Large multimodal models (LMMs) are increasingly adopted as judges in multimodal evaluation systems due to their strong instruction following and consistency with human preferences. However, their ability to follow diverse, fine-grained evaluation criteria remains underexplored. We develop Multi-Crit, a benchmark for evaluating multimodal judges on their capacity to follow pluralistic criteria and produce reliable criterion-level judgments. Covering both open-ended generation and verifiable reasoning tasks, Multi-Crit is built through a rigorous data curation pipeline that gathers challenging response pairs with multi-criterion human annotations. It further introduces three novel metrics for systematically assessing pluralistic adherence, criterion-switching flexibility, and the ability to recognize criterion-level preference conflicts. Comprehensive analysis of 25 LMMs reveals that 1) proprietary models still struggle to maintain consistent adherence to pluralistic criteria--especially in open-ended evaluation; 2) open-source models lag further behind in flexibly following diverse criteria; and 3) critic fine-tuning with holistic judgment signals enhances visual grounding but fails to generalize to pluralistic criterion-level judgment. Additional analyses on reasoning fine-tuning, test-time scaling, and boundary consistency between open-source and proprietary models further probe the limits of current multimodal judges. As a pioneering study, Multi-Crit lays the foundation for building reliable and steerable multimodal AI evaluation. |
| title | Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Following |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2511.21662 |