UmniBench: Unified Understand and Generation Model Oriented Omni-dimensional Benchmark
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911327618859008 |
|---|---|
| author | Liu, Kai Chen, Leyang Li, Wenbo Chen, Zhikai Wang, Zhixin Pei, Renjing Kong, Linghe Zhang, Yulun |
| author_facet | Liu, Kai Chen, Leyang Li, Wenbo Chen, Zhikai Wang, Zhixin Pei, Renjing Kong, Linghe Zhang, Yulun |
| contents | Unifying multimodal understanding and generation has shown impressive capabilities in cutting-edge proprietary systems. However, evaluations of unified multimodal models (UMMs) remain decoupled, assessing their understanding and generation abilities separately with corresponding datasets. To address this, we propose UmniBench, a benchmark tailored for UMMs with omni-dimensional evaluation. First, UmniBench can assess the understanding, generation, and editing ability within a single evaluation process. Based on human-examined prompts and QA pairs, UmniBench leverages UMM itself to evaluate its generation and editing ability with its understanding ability. This simple but effective paradigm allows comprehensive evaluation of UMMs. Second, UmniBench covers 13 major domains and more than 200 concepts, ensuring a thorough inspection of UMMs. Moreover, UmniBench can also decouple and separately evaluate understanding, generation, and editing abilities, providing a fine-grained assessment. Based on UmniBench, we benchmark 24 popular models, including both UMMs and single-ability large models. We hope this benchmark provides a more comprehensive and objective view of unified models and logistical support for improving the performance of the community model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_17196 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | UmniBench: Unified Understand and Generation Model Oriented Omni-dimensional Benchmark Liu, Kai Chen, Leyang Li, Wenbo Chen, Zhikai Wang, Zhixin Pei, Renjing Kong, Linghe Zhang, Yulun Artificial Intelligence Unifying multimodal understanding and generation has shown impressive capabilities in cutting-edge proprietary systems. However, evaluations of unified multimodal models (UMMs) remain decoupled, assessing their understanding and generation abilities separately with corresponding datasets. To address this, we propose UmniBench, a benchmark tailored for UMMs with omni-dimensional evaluation. First, UmniBench can assess the understanding, generation, and editing ability within a single evaluation process. Based on human-examined prompts and QA pairs, UmniBench leverages UMM itself to evaluate its generation and editing ability with its understanding ability. This simple but effective paradigm allows comprehensive evaluation of UMMs. Second, UmniBench covers 13 major domains and more than 200 concepts, ensuring a thorough inspection of UMMs. Moreover, UmniBench can also decouple and separately evaluate understanding, generation, and editing abilities, providing a fine-grained assessment. Based on UmniBench, we benchmark 24 popular models, including both UMMs and single-ability large models. We hope this benchmark provides a more comprehensive and objective view of unified models and logistical support for improving the performance of the community model. |
| title | UmniBench: Unified Understand and Generation Model Oriented Omni-dimensional Benchmark |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2512.17196 |