Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.14478 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913745478877184 |
|---|---|
| author | Fang, Xinyu Chen, Zhijian Lan, Kai Ma, Lixin Ding, Shengyuan Liang, Yingji Zhao, Xiangyu Wen, Farong Zhang, Zicheng Zhang, Guofeng Duan, Haodong Chen, Kai Lin, Dahua |
| author_facet | Fang, Xinyu Chen, Zhijian Lan, Kai Ma, Lixin Ding, Shengyuan Liang, Yingji Zhao, Xiangyu Wen, Farong Zhang, Zicheng Zhang, Guofeng Duan, Haodong Chen, Kai Lin, Dahua |
| contents | Creativity is a fundamental aspect of intelligence, involving the ability to generate novel and appropriate solutions across diverse contexts. While Large Language Models (LLMs) have been extensively evaluated for their creative capabilities, the assessment of Multimodal Large Language Models (MLLMs) in this domain remains largely unexplored. To address this gap, we introduce Creation-MMBench, a multimodal benchmark specifically designed to evaluate the creative capabilities of MLLMs in real-world, image-based tasks. The benchmark comprises 765 test cases spanning 51 fine-grained tasks. To ensure rigorous evaluation, we define instance-specific evaluation criteria for each test case, guiding the assessment of both general response quality and factual consistency with visual inputs. Experimental results reveal that current open-source MLLMs significantly underperform compared to proprietary models in creative tasks. Furthermore, our analysis demonstrates that visual fine-tuning can negatively impact the base LLM's creative abilities. Creation-MMBench provides valuable insights for advancing MLLM creativity and establishes a foundation for future improvements in multimodal generative intelligence. Full data and evaluation code is released on https://github.com/open-compass/Creation-MMBench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_14478 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLM Fang, Xinyu Chen, Zhijian Lan, Kai Ma, Lixin Ding, Shengyuan Liang, Yingji Zhao, Xiangyu Wen, Farong Zhang, Zicheng Zhang, Guofeng Duan, Haodong Chen, Kai Lin, Dahua Computer Vision and Pattern Recognition Creativity is a fundamental aspect of intelligence, involving the ability to generate novel and appropriate solutions across diverse contexts. While Large Language Models (LLMs) have been extensively evaluated for their creative capabilities, the assessment of Multimodal Large Language Models (MLLMs) in this domain remains largely unexplored. To address this gap, we introduce Creation-MMBench, a multimodal benchmark specifically designed to evaluate the creative capabilities of MLLMs in real-world, image-based tasks. The benchmark comprises 765 test cases spanning 51 fine-grained tasks. To ensure rigorous evaluation, we define instance-specific evaluation criteria for each test case, guiding the assessment of both general response quality and factual consistency with visual inputs. Experimental results reveal that current open-source MLLMs significantly underperform compared to proprietary models in creative tasks. Furthermore, our analysis demonstrates that visual fine-tuning can negatively impact the base LLM's creative abilities. Creation-MMBench provides valuable insights for advancing MLLM creativity and establishes a foundation for future improvements in multimodal generative intelligence. Full data and evaluation code is released on https://github.com/open-compass/Creation-MMBench. |
| title | Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLM |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2503.14478 |