UNICBench: UNIfied Counting Benchmark for MLLM
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912934300483584 |
|---|---|
| author | Rong, Chenggang Han, Tao Zhao, Zhiyuan Fan, Yaowu Wan, Jia Guo, Song Yuan, Yuan Gao, Junyu |
| author_facet | Rong, Chenggang Han, Tao Zhao, Zhiyuan Fan, Yaowu Wan, Jia Guo, Song Yuan, Yuan Gao, Junyu |
| contents | Counting is a core capability for multimodal large language models (MLLMs), yet there is no unified counting dataset to rigorously evaluate this ability across image, text, and audio. We present UNICBench, a unified multimodal, multi level counting benchmark and evaluation toolkit with accurate ground truth, deterministic numeric parsing, and stratified reporting. The corpus comprises 5,300 images (5,508 QA), 872 documents (5,888 QA), and 2,069 audio clips (2,905 QA), annotated with a three level capability taxonomy and difficulty tags. Under a standardized protocol with fixed splits/prompts/seeds and modality specific matching rules, we evaluate 45 state-of-the-art MLLMs across modalities. Results show strong performance on some basic counting tasks but significant gaps on reasoning and the hardest partitions, highlighting long-tail errors and substantial headroom for improving general counting. UNICBench offers a rigorous and comparable basis for measurement and a public toolkit to accelerate progress. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_00595 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | UNICBench: UNIfied Counting Benchmark for MLLM Rong, Chenggang Han, Tao Zhao, Zhiyuan Fan, Yaowu Wan, Jia Guo, Song Yuan, Yuan Gao, Junyu Computer Vision and Pattern Recognition Counting is a core capability for multimodal large language models (MLLMs), yet there is no unified counting dataset to rigorously evaluate this ability across image, text, and audio. We present UNICBench, a unified multimodal, multi level counting benchmark and evaluation toolkit with accurate ground truth, deterministic numeric parsing, and stratified reporting. The corpus comprises 5,300 images (5,508 QA), 872 documents (5,888 QA), and 2,069 audio clips (2,905 QA), annotated with a three level capability taxonomy and difficulty tags. Under a standardized protocol with fixed splits/prompts/seeds and modality specific matching rules, we evaluate 45 state-of-the-art MLLMs across modalities. Results show strong performance on some basic counting tasks but significant gaps on reasoning and the hardest partitions, highlighting long-tail errors and substantial headroom for improving general counting. UNICBench offers a rigorous and comparable basis for measurement and a public toolkit to accelerate progress. |
| title | UNICBench: UNIfied Counting Benchmark for MLLM |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2603.00595 |