MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914957466009600 |
|---|---|
| author | Li, Kaixin Tian, Yuchen Hu, Qisheng Luo, Ziyang Huang, Zhiyong Ma, Jing |
| author_facet | Li, Kaixin Tian, Yuchen Hu, Qisheng Luo, Ziyang Huang, Zhiyong Ma, Jing |
| contents | Programming often involves converting detailed and complex specifications into code, a process during which developers typically utilize visual aids to more effectively convey concepts. While recent developments in Large Multimodal Models have demonstrated remarkable abilities in visual reasoning and mathematical tasks, there is little work on investigating whether these models can effectively interpret visual elements for code generation. To this end, we present MMCode, the first multi-modal coding dataset for evaluating algorithmic problem-solving skills in visually rich contexts. MMCode contains 3,548 questions and 6,620 images collected from real-world programming challenges harvested from 10 code competition websites, presenting significant challenges due to the extreme demand for reasoning abilities. Our experiment results show that current state-of-the-art models struggle to solve these problems. The results highlight the lack of powerful vision-code models, and we hope MMCode can serve as an inspiration for future works in this domain. The data and code are publicly available at https://github.com/likaixin2000/MMCode. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_09486 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems Li, Kaixin Tian, Yuchen Hu, Qisheng Luo, Ziyang Huang, Zhiyong Ma, Jing Computation and Language Computer Vision and Pattern Recognition Software Engineering Programming often involves converting detailed and complex specifications into code, a process during which developers typically utilize visual aids to more effectively convey concepts. While recent developments in Large Multimodal Models have demonstrated remarkable abilities in visual reasoning and mathematical tasks, there is little work on investigating whether these models can effectively interpret visual elements for code generation. To this end, we present MMCode, the first multi-modal coding dataset for evaluating algorithmic problem-solving skills in visually rich contexts. MMCode contains 3,548 questions and 6,620 images collected from real-world programming challenges harvested from 10 code competition websites, presenting significant challenges due to the extreme demand for reasoning abilities. Our experiment results show that current state-of-the-art models struggle to solve these problems. The results highlight the lack of powerful vision-code models, and we hope MMCode can serve as an inspiration for future works in this domain. The data and code are publicly available at https://github.com/likaixin2000/MMCode. |
| title | MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems |
| topic | Computation and Language Computer Vision and Pattern Recognition Software Engineering |
| url | https://arxiv.org/abs/2404.09486 |