Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2406.09961 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913711823781888 |
|---|---|
| author | Yang, Cheng Shi, Chufan Liu, Yaxin Shui, Bo Wang, Junjie Jing, Mohan Xu, Linran Zhu, Xinyu Li, Siheng Zhang, Yuxiang Liu, Gongye Nie, Xiaomei Cai, Deng Yang, Yujiu |
| author_facet | Yang, Cheng Shi, Chufan Liu, Yaxin Shui, Bo Wang, Junjie Jing, Mohan Xu, Linran Zhu, Xinyu Li, Siheng Zhang, Yuxiang Liu, Gongye Nie, Xiaomei Cai, Deng Yang, Yujiu |
| contents | We introduce a new benchmark, ChartMimic, aimed at assessing the visually-grounded code generation capabilities of large multimodal models (LMMs). ChartMimic utilizes information-intensive visual charts and textual instructions as inputs, requiring LMMs to generate the corresponding code for chart rendering. ChartMimic includes 4,800 human-curated (figure, instruction, code) triplets, which represent the authentic chart use cases found in scientific papers across various domains (e.g., Physics, Computer Science, Economics, etc). These charts span 18 regular types and 4 advanced types, diversifying into 201 subcategories. Furthermore, we propose multi-level evaluation metrics to provide an automatic and thorough assessment of the output code and the rendered charts. Unlike existing code generation benchmarks, ChartMimic places emphasis on evaluating LMMs' capacity to harmonize a blend of cognitive capabilities, encompassing visual understanding, code generation, and cross-modal reasoning. The evaluation of $3$ proprietary models and 14 open-weight models highlights the substantial challenges posed by ChartMimic. Even the advanced GPT-4o, InternVL2-Llama3-76B only achieved an average score across Direct Mimic and Customized Mimic tasks of 82.2 and 61.6, respectively, indicating significant room for improvement. We anticipate that ChartMimic will inspire the development of LMMs, advancing the pursuit of artificial general intelligence. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_09961 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation Yang, Cheng Shi, Chufan Liu, Yaxin Shui, Bo Wang, Junjie Jing, Mohan Xu, Linran Zhu, Xinyu Li, Siheng Zhang, Yuxiang Liu, Gongye Nie, Xiaomei Cai, Deng Yang, Yujiu Software Engineering Computation and Language Computer Vision and Pattern Recognition We introduce a new benchmark, ChartMimic, aimed at assessing the visually-grounded code generation capabilities of large multimodal models (LMMs). ChartMimic utilizes information-intensive visual charts and textual instructions as inputs, requiring LMMs to generate the corresponding code for chart rendering. ChartMimic includes 4,800 human-curated (figure, instruction, code) triplets, which represent the authentic chart use cases found in scientific papers across various domains (e.g., Physics, Computer Science, Economics, etc). These charts span 18 regular types and 4 advanced types, diversifying into 201 subcategories. Furthermore, we propose multi-level evaluation metrics to provide an automatic and thorough assessment of the output code and the rendered charts. Unlike existing code generation benchmarks, ChartMimic places emphasis on evaluating LMMs' capacity to harmonize a blend of cognitive capabilities, encompassing visual understanding, code generation, and cross-modal reasoning. The evaluation of $3$ proprietary models and 14 open-weight models highlights the substantial challenges posed by ChartMimic. Even the advanced GPT-4o, InternVL2-Llama3-76B only achieved an average score across Direct Mimic and Customized Mimic tasks of 82.2 and 61.6, respectively, indicating significant room for improvement. We anticipate that ChartMimic will inspire the development of LMMs, advancing the pursuit of artificial general intelligence. |
| title | ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation |
| topic | Software Engineering Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.09961 |