Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2506.14028 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917004100763648 |
|---|---|
| author | Peng, Xueqing Qian, Lingfei Wang, Yan Xiang, Ruoyu He, Yueru Ren, Yang Jiang, Mingyang Zhang, Vincent Jim Guo, Yuqing Zhao, Jeff He, Huan Han, Yi Feng, Yun Jiang, Yuechen Cao, Yupeng Li, Haohang Yu, Yangyang Wang, Xiaoyu Gao, Penglei Lin, Shengyuan Wang, Keyi Yang, Shanshan Zhao, Yilun Liu, Zhiwei Lu, Peng Huang, Jerry Wang, Suyuchen Papadopoulos, Triantafillos Giannouris, Polydoros Soufleri, Efstathia Chen, Nuo Deng, Zhiyang Fu, Heming Zhao, Yijia Lin, Mingquan Qiu, Meikang Smith, Kaleb E Cohan, Arman Liu, Xiao-Yang Huang, Jimin Xiong, Guojun Lopez-Lira, Alejandro Chen, Xi Tsujii, Junichi Nie, Jian-Yun Ananiadou, Sophia Xie, Qianqian |
| author_facet | Peng, Xueqing Qian, Lingfei Wang, Yan Xiang, Ruoyu He, Yueru Ren, Yang Jiang, Mingyang Zhang, Vincent Jim Guo, Yuqing Zhao, Jeff He, Huan Han, Yi Feng, Yun Jiang, Yuechen Cao, Yupeng Li, Haohang Yu, Yangyang Wang, Xiaoyu Gao, Penglei Lin, Shengyuan Wang, Keyi Yang, Shanshan Zhao, Yilun Liu, Zhiwei Lu, Peng Huang, Jerry Wang, Suyuchen Papadopoulos, Triantafillos Giannouris, Polydoros Soufleri, Efstathia Chen, Nuo Deng, Zhiyang Fu, Heming Zhao, Yijia Lin, Mingquan Qiu, Meikang Smith, Kaleb E Cohan, Arman Liu, Xiao-Yang Huang, Jimin Xiong, Guojun Lopez-Lira, Alejandro Chen, Xi Tsujii, Junichi Nie, Jian-Yun Ananiadou, Sophia Xie, Qianqian |
| contents | Real-world financial analysis involves information across multiple languages and modalities, from reports and news to scanned filings and meeting recordings. Yet most existing evaluations of LLMs in finance remain text-only, monolingual, and largely saturated by current models. To bridge these gaps, we present MultiFinBen, the first expert-annotated multilingual (five languages) and multimodal (text, vision, audio) benchmark for evaluating LLMs in realistic financial contexts. MultiFinBen introduces two new task families: multilingual financial reasoning, which tests cross-lingual evidence integration from filings and news, and financial OCR, which extracts structured text from scanned documents containing tables and charts. Rather than aggregating all available datasets, we apply a structured, difficulty-aware selection based on advanced model performance, ensuring balanced challenge and removing redundant tasks. Evaluating 21 leading LLMs shows that even frontier multimodal models like GPT-4o achieve only 46.01% overall, stronger on vision and audio but dropping sharply in multilingual settings. These findings expose persistent limitations in multilingual, multimodal, and expert-level financial reasoning. All datasets, evaluation scripts, and leaderboards are publicly released. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_14028 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Application Peng, Xueqing Qian, Lingfei Wang, Yan Xiang, Ruoyu He, Yueru Ren, Yang Jiang, Mingyang Zhang, Vincent Jim Guo, Yuqing Zhao, Jeff He, Huan Han, Yi Feng, Yun Jiang, Yuechen Cao, Yupeng Li, Haohang Yu, Yangyang Wang, Xiaoyu Gao, Penglei Lin, Shengyuan Wang, Keyi Yang, Shanshan Zhao, Yilun Liu, Zhiwei Lu, Peng Huang, Jerry Wang, Suyuchen Papadopoulos, Triantafillos Giannouris, Polydoros Soufleri, Efstathia Chen, Nuo Deng, Zhiyang Fu, Heming Zhao, Yijia Lin, Mingquan Qiu, Meikang Smith, Kaleb E Cohan, Arman Liu, Xiao-Yang Huang, Jimin Xiong, Guojun Lopez-Lira, Alejandro Chen, Xi Tsujii, Junichi Nie, Jian-Yun Ananiadou, Sophia Xie, Qianqian Computation and Language Real-world financial analysis involves information across multiple languages and modalities, from reports and news to scanned filings and meeting recordings. Yet most existing evaluations of LLMs in finance remain text-only, monolingual, and largely saturated by current models. To bridge these gaps, we present MultiFinBen, the first expert-annotated multilingual (five languages) and multimodal (text, vision, audio) benchmark for evaluating LLMs in realistic financial contexts. MultiFinBen introduces two new task families: multilingual financial reasoning, which tests cross-lingual evidence integration from filings and news, and financial OCR, which extracts structured text from scanned documents containing tables and charts. Rather than aggregating all available datasets, we apply a structured, difficulty-aware selection based on advanced model performance, ensuring balanced challenge and removing redundant tasks. Evaluating 21 leading LLMs shows that even frontier multimodal models like GPT-4o achieve only 46.01% overall, stronger on vision and audio but dropping sharply in multilingual settings. These findings expose persistent limitations in multilingual, multimodal, and expert-level financial reasoning. All datasets, evaluation scripts, and leaderboards are publicly released. |
| title | MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Application |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.14028 |