ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detection
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911604800487424 |
|---|---|
| author | Yan, Yibo Wang, Shen Huo, Jiahao Li, Hang Li, Boyan Su, Jiamin Gao, Xiong Zhang, Yi-Fan Xu, Tianlong Chu, Zhendong Zhong, Aoxiao Wang, Kun Xiong, Hui Yu, Philip S. Hu, Xuming Wen, Qingsong |
| author_facet | Yan, Yibo Wang, Shen Huo, Jiahao Li, Hang Li, Boyan Su, Jiamin Gao, Xiong Zhang, Yi-Fan Xu, Tianlong Chu, Zhendong Zhong, Aoxiao Wang, Kun Xiong, Hui Yu, Philip S. Hu, Xuming Wen, Qingsong |
| contents | As the field of Multimodal Large Language Models (MLLMs) continues to evolve, their potential to revolutionize artificial intelligence is particularly promising, especially in addressing mathematical reasoning tasks. Current mathematical benchmarks predominantly focus on evaluating MLLMs' problem-solving ability, yet there is a crucial gap in addressing more complex scenarios such as error detection, for enhancing reasoning capability in complicated settings. To fill this gap, we formally formulate the new task: multimodal error detection, and introduce ErrorRadar, the first benchmark designed to assess MLLMs' capabilities in such a task. ErrorRadar evaluates two sub-tasks: error step identification and error categorization, providing a comprehensive framework for evaluating MLLMs' complex mathematical reasoning ability. It consists of 2,500 high-quality multimodal K-12 mathematical problems, collected from real-world student interactions in an educational organization, with rigorous annotation and rich metadata such as problem type and error category. Through extensive experiments, we evaluated both open-source and closed-source representative MLLMs, benchmarking their performance against educational expert evaluators. Results indicate significant challenges still remain, as GPT-4o with best performance is still around 10% behind human evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_04509 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detection Yan, Yibo Wang, Shen Huo, Jiahao Li, Hang Li, Boyan Su, Jiamin Gao, Xiong Zhang, Yi-Fan Xu, Tianlong Chu, Zhendong Zhong, Aoxiao Wang, Kun Xiong, Hui Yu, Philip S. Hu, Xuming Wen, Qingsong Computation and Language As the field of Multimodal Large Language Models (MLLMs) continues to evolve, their potential to revolutionize artificial intelligence is particularly promising, especially in addressing mathematical reasoning tasks. Current mathematical benchmarks predominantly focus on evaluating MLLMs' problem-solving ability, yet there is a crucial gap in addressing more complex scenarios such as error detection, for enhancing reasoning capability in complicated settings. To fill this gap, we formally formulate the new task: multimodal error detection, and introduce ErrorRadar, the first benchmark designed to assess MLLMs' capabilities in such a task. ErrorRadar evaluates two sub-tasks: error step identification and error categorization, providing a comprehensive framework for evaluating MLLMs' complex mathematical reasoning ability. It consists of 2,500 high-quality multimodal K-12 mathematical problems, collected from real-world student interactions in an educational organization, with rigorous annotation and rich metadata such as problem type and error category. Through extensive experiments, we evaluated both open-source and closed-source representative MLLMs, benchmarking their performance against educational expert evaluators. Results indicate significant challenges still remain, as GPT-4o with best performance is still around 10% behind human evaluation. |
| title | ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detection |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.04509 |