Code-Vision: Evaluating Multimodal LLMs Logic Understanding and Code Generation Capabilities
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916618026614784 |
|---|---|
| author | Wang, Hanbin Zhou, Xiaoxuan Xu, Zhipeng Cheng, Keyuan Zuo, Yuxin Tian, Kai Song, Jingwei Lu, Junting Hu, Wenhui Liu, Xueyang |
| author_facet | Wang, Hanbin Zhou, Xiaoxuan Xu, Zhipeng Cheng, Keyuan Zuo, Yuxin Tian, Kai Song, Jingwei Lu, Junting Hu, Wenhui Liu, Xueyang |
| contents | This paper introduces Code-Vision, a benchmark designed to evaluate the logical understanding and code generation capabilities of Multimodal Large Language Models (MLLMs). It challenges MLLMs to generate a correct program that fulfills specific functionality requirements based on a given flowchart, which visually represents the desired algorithm or process. Code-Vision comprises three subsets: HumanEval-V, Algorithm, and MATH, which evaluate MLLMs' coding abilities across basic programming, algorithmic, and mathematical problem-solving domains. Our experiments evaluate 12 MLLMs on Code-Vision. Experimental results demonstrate that there is a large performance difference between proprietary and open-source models. On Hard problems, GPT-4o can achieve 79.3% pass@1, but the best open-source model only achieves 15%. Further experiments reveal that Code-Vision can pose unique challenges compared to other multimodal reasoning benchmarks MMCode and MathVista. We also explore the reason for the poor performance of the open-source models. All data and codes are available at https://github.com/wanghanbinpanda/CodeVision. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_11829 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Code-Vision: Evaluating Multimodal LLMs Logic Understanding and Code Generation Capabilities Wang, Hanbin Zhou, Xiaoxuan Xu, Zhipeng Cheng, Keyuan Zuo, Yuxin Tian, Kai Song, Jingwei Lu, Junting Hu, Wenhui Liu, Xueyang Computation and Language Artificial Intelligence Software Engineering This paper introduces Code-Vision, a benchmark designed to evaluate the logical understanding and code generation capabilities of Multimodal Large Language Models (MLLMs). It challenges MLLMs to generate a correct program that fulfills specific functionality requirements based on a given flowchart, which visually represents the desired algorithm or process. Code-Vision comprises three subsets: HumanEval-V, Algorithm, and MATH, which evaluate MLLMs' coding abilities across basic programming, algorithmic, and mathematical problem-solving domains. Our experiments evaluate 12 MLLMs on Code-Vision. Experimental results demonstrate that there is a large performance difference between proprietary and open-source models. On Hard problems, GPT-4o can achieve 79.3% pass@1, but the best open-source model only achieves 15%. Further experiments reveal that Code-Vision can pose unique challenges compared to other multimodal reasoning benchmarks MMCode and MathVista. We also explore the reason for the poor performance of the open-source models. All data and codes are available at https://github.com/wanghanbinpanda/CodeVision. |
| title | Code-Vision: Evaluating Multimodal LLMs Logic Understanding and Code Generation Capabilities |
| topic | Computation and Language Artificial Intelligence Software Engineering |
| url | https://arxiv.org/abs/2502.11829 |