FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909629445832704 |
|---|---|
| author | Luo, Junyu Kou, Zhizhuo Yang, Liming Luo, Xiao Huang, Jinsheng Xiao, Zhiping Peng, Jingshu Liu, Chengzhong Ji, Jiaming Liu, Xuanzhe Han, Sirui Zhang, Ming Guo, Yike |
| author_facet | Luo, Junyu Kou, Zhizhuo Yang, Liming Luo, Xiao Huang, Jinsheng Xiao, Zhiping Peng, Jingshu Liu, Chengzhong Ji, Jiaming Liu, Xuanzhe Han, Sirui Zhang, Ming Guo, Yike |
| contents | Multimodal Large Language Models (MLLMs) have experienced rapid development in recent years. However, in the financial domain, there is a notable lack of effective and specialized multimodal evaluation datasets. To advance the development of MLLMs in the finance domain, we introduce FinMME, encompassing more than 11,000 high-quality financial research samples across 18 financial domains and 6 asset classes, featuring 10 major chart types and 21 subtypes. We ensure data quality through 20 annotators and carefully designed validation mechanisms. Additionally, we develop FinScore, an evaluation system incorporating hallucination penalties and multi-dimensional capability assessment to provide an unbiased evaluation. Extensive experimental results demonstrate that even state-of-the-art models like GPT-4o exhibit unsatisfactory performance on FinMME, highlighting its challenging nature. The benchmark exhibits high robustness with prediction variations under different prompts remaining below 1%, demonstrating superior reliability compared to existing datasets. Our dataset and evaluation protocol are available at https://huggingface.co/datasets/luojunyu/FinMME and https://github.com/luo-junyu/FinMME. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_24714 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation Luo, Junyu Kou, Zhizhuo Yang, Liming Luo, Xiao Huang, Jinsheng Xiao, Zhiping Peng, Jingshu Liu, Chengzhong Ji, Jiaming Liu, Xuanzhe Han, Sirui Zhang, Ming Guo, Yike Computation and Language Multimodal Large Language Models (MLLMs) have experienced rapid development in recent years. However, in the financial domain, there is a notable lack of effective and specialized multimodal evaluation datasets. To advance the development of MLLMs in the finance domain, we introduce FinMME, encompassing more than 11,000 high-quality financial research samples across 18 financial domains and 6 asset classes, featuring 10 major chart types and 21 subtypes. We ensure data quality through 20 annotators and carefully designed validation mechanisms. Additionally, we develop FinScore, an evaluation system incorporating hallucination penalties and multi-dimensional capability assessment to provide an unbiased evaluation. Extensive experimental results demonstrate that even state-of-the-art models like GPT-4o exhibit unsatisfactory performance on FinMME, highlighting its challenging nature. The benchmark exhibits high robustness with prediction variations under different prompts remaining below 1%, demonstrating superior reliability compared to existing datasets. Our dataset and evaluation protocol are available at https://huggingface.co/datasets/luojunyu/FinMME and https://github.com/luo-junyu/FinMME. |
| title | FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.24714 |