Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.24897 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911531144314880 |
|---|---|
| author | Shi, Yang Dong, Yuhao Ding, Yue Wang, Yuran Zhu, Xuanyu Zhou, Sheng Liu, Wenting Tian, Haochen Wang, Rundong Wang, Huanqian Liu, Zuyan Zeng, Bohan Chen, Ruizhe Wang, Qixun Zhang, Zhuoran Chen, Xinlong Tong, Chengzhuo Li, Bozhou Liu, Qiang Wang, Haotian Yang, Wenjing Zhang, Yuanxing Wan, Pengfei Zhang, Yi-Fan Liu, Ziwei |
| author_facet | Shi, Yang Dong, Yuhao Ding, Yue Wang, Yuran Zhu, Xuanyu Zhou, Sheng Liu, Wenting Tian, Haochen Wang, Rundong Wang, Huanqian Liu, Zuyan Zeng, Bohan Chen, Ruizhe Wang, Qixun Zhang, Zhuoran Chen, Xinlong Tong, Chengzhuo Li, Bozhou Liu, Qiang Wang, Haotian Yang, Wenjing Zhang, Yuanxing Wan, Pengfei Zhang, Yi-Fan Liu, Ziwei |
| contents | The integration of visual understanding and generation into unified multimodal models represents a significant stride toward general-purpose AI. However, a fundamental question remains unanswered by existing benchmarks: does this architectural unification actually enable synergetic interaction between the constituent capabilities? Existing evaluation paradigms, which primarily assess understanding and generation in isolation, are insufficient for determining whether a unified model can leverage its understanding to enhance its generation, or use generative simulation to facilitate deeper comprehension. To address this critical gap, we introduce RealUnify, a benchmark specifically designed to evaluate bidirectional capability synergy. RealUnify comprises 1,000 meticulously human-annotated instances spanning 10 categories and 32 subtasks. It is structured around two core axes: 1) Understanding Enhances Generation, which requires reasoning (e.g., commonsense, logic) to guide image generation, and 2) Generation Enhances Understanding, which necessitates mental simulation or reconstruction (e.g., of transformed or disordered visual inputs) to solve reasoning tasks. A key contribution is our dual-evaluation protocol, which combines direct end-to-end assessment with a diagnostic stepwise evaluation that decomposes tasks into distinct understanding and generation phases. This protocol allows us to precisely discern whether performance bottlenecks stem from deficiencies in core abilities or from a failure to integrate them. Through large-scale evaluations of 12 leading unified models and 6 specialized baselines, we find that current unified models still struggle to achieve effective synergy, indicating that architectural unification alone is insufficient. These results highlight the need for new training strategies and inductive biases to fully unlock the potential of unified modeling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_24897 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmark Shi, Yang Dong, Yuhao Ding, Yue Wang, Yuran Zhu, Xuanyu Zhou, Sheng Liu, Wenting Tian, Haochen Wang, Rundong Wang, Huanqian Liu, Zuyan Zeng, Bohan Chen, Ruizhe Wang, Qixun Zhang, Zhuoran Chen, Xinlong Tong, Chengzhuo Li, Bozhou Liu, Qiang Wang, Haotian Yang, Wenjing Zhang, Yuanxing Wan, Pengfei Zhang, Yi-Fan Liu, Ziwei Artificial Intelligence The integration of visual understanding and generation into unified multimodal models represents a significant stride toward general-purpose AI. However, a fundamental question remains unanswered by existing benchmarks: does this architectural unification actually enable synergetic interaction between the constituent capabilities? Existing evaluation paradigms, which primarily assess understanding and generation in isolation, are insufficient for determining whether a unified model can leverage its understanding to enhance its generation, or use generative simulation to facilitate deeper comprehension. To address this critical gap, we introduce RealUnify, a benchmark specifically designed to evaluate bidirectional capability synergy. RealUnify comprises 1,000 meticulously human-annotated instances spanning 10 categories and 32 subtasks. It is structured around two core axes: 1) Understanding Enhances Generation, which requires reasoning (e.g., commonsense, logic) to guide image generation, and 2) Generation Enhances Understanding, which necessitates mental simulation or reconstruction (e.g., of transformed or disordered visual inputs) to solve reasoning tasks. A key contribution is our dual-evaluation protocol, which combines direct end-to-end assessment with a diagnostic stepwise evaluation that decomposes tasks into distinct understanding and generation phases. This protocol allows us to precisely discern whether performance bottlenecks stem from deficiencies in core abilities or from a failure to integrate them. Through large-scale evaluations of 12 leading unified models and 6 specialized baselines, we find that current unified models still struggle to achieve effective synergy, indicating that architectural unification alone is insufficient. These results highlight the need for new training strategies and inductive biases to fully unlock the potential of unified modeling. |
| title | RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmark |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2509.24897 |