SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmark
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929237675474944 |
|---|---|
| author | Liang, Zhenwen Guo, Kehan Liu, Gang Guo, Taicheng Zhou, Yujun Yang, Tianyu Jiao, Jiajun Pi, Renjie Zhang, Jipeng Zhang, Xiangliang |
| author_facet | Liang, Zhenwen Guo, Kehan Liu, Gang Guo, Taicheng Zhou, Yujun Yang, Tianyu Jiao, Jiajun Pi, Renjie Zhang, Jipeng Zhang, Xiangliang |
| contents | The paper introduces SceMQA, a novel benchmark for scientific multimodal question answering at the college entrance level. It addresses a critical educational phase often overlooked in existing benchmarks, spanning high school to pre-college levels. SceMQA focuses on core science subjects including Mathematics, Physics, Chemistry, and Biology. It features a blend of multiple-choice and free-response formats, ensuring a comprehensive evaluation of AI models' abilities. Additionally, our benchmark provides specific knowledge points for each problem and detailed explanations for each answer. SceMQA also uniquely presents problems with identical contexts but varied questions to facilitate a more thorough and accurate assessment of reasoning capabilities. In the experiment, we evaluate both open-source and close-source state-of-the-art Multimodal Large Language Models (MLLMs), across various experimental settings. The results show that further research and development are needed in developing more capable MLLM, as highlighted by only 50% to 60% accuracy achieved by the strongest models. Our benchmark and analysis will be available at https://scemqa.github.io/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_05138 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmark Liang, Zhenwen Guo, Kehan Liu, Gang Guo, Taicheng Zhou, Yujun Yang, Tianyu Jiao, Jiajun Pi, Renjie Zhang, Jipeng Zhang, Xiangliang Artificial Intelligence Computation and Language The paper introduces SceMQA, a novel benchmark for scientific multimodal question answering at the college entrance level. It addresses a critical educational phase often overlooked in existing benchmarks, spanning high school to pre-college levels. SceMQA focuses on core science subjects including Mathematics, Physics, Chemistry, and Biology. It features a blend of multiple-choice and free-response formats, ensuring a comprehensive evaluation of AI models' abilities. Additionally, our benchmark provides specific knowledge points for each problem and detailed explanations for each answer. SceMQA also uniquely presents problems with identical contexts but varied questions to facilitate a more thorough and accurate assessment of reasoning capabilities. In the experiment, we evaluate both open-source and close-source state-of-the-art Multimodal Large Language Models (MLLMs), across various experimental settings. The results show that further research and development are needed in developing more capable MLLM, as highlighted by only 50% to 60% accuracy achieved by the strongest models. Our benchmark and analysis will be available at https://scemqa.github.io/ |
| title | SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmark |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2402.05138 |