FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915497454338048 |
|---|---|
| author | Hu, Liang Jiao, Jianpeng Liu, Jiashuo Ren, Yanle Wen, Zhoufutu Zhang, Kaiyuan Zhang, Xuanliang Gao, Xiang He, Tianci Hu, Fei Liao, Yali Wang, Zaiyuan Yang, Chenghao Yang, Qianyu Yin, Mingren Zeng, Zhiyuan Zhang, Ge Zhang, Xinyi Zhao, Xiying Zhu, Zhenwei Namkoong, Hongseok Huang, Wenhao Tang, Yuwen |
| author_facet | Hu, Liang Jiao, Jianpeng Liu, Jiashuo Ren, Yanle Wen, Zhoufutu Zhang, Kaiyuan Zhang, Xuanliang Gao, Xiang He, Tianci Hu, Fei Liao, Yali Wang, Zaiyuan Yang, Chenghao Yang, Qianyu Yin, Mingren Zeng, Zhiyuan Zhang, Ge Zhang, Xinyi Zhao, Xiying Zhu, Zhenwei Namkoong, Hongseok Huang, Wenhao Tang, Yuwen |
| contents | Search has emerged as core infrastructure for LLM-based agents and is widely viewed as critical on the path toward more general intelligence. Finance is a particularly demanding proving ground: analysts routinely conduct complex, multi-step searches over time-sensitive, domain-specific data, making it ideal for assessing both search proficiency and knowledge-grounded reasoning. Yet no existing open financial datasets evaluate data searching capability of end-to-end agents, largely because constructing realistic, complicated tasks requires deep financial expertise and time-sensitive data is hard to evaluate. We present FinSearchComp, the first fully open-source agent benchmark for realistic, open-domain financial search and reasoning. FinSearchComp comprises three tasks -- Time-Sensitive Data Fetching, Simple Historical Lookup, and Complex Historical Investigation -- closely reproduce real-world financial analyst workflows. To ensure difficulty and reliability, we engage 70 professional financial experts for annotation and implement a rigorous multi-stage quality-assurance pipeline. The benchmark includes 635 questions spanning global and Greater China markets, and we evaluate 21 models (products) on it. Grok 4 (web) tops the global subset, approaching expert-level accuracy. DouBao (web) leads on the Greater China subset. Experimental analyses show that equipping agents with web search and financial plugins substantially improves results on FinSearchComp, and the country origin of models and tools impact performance significantly.By aligning with realistic analyst tasks and providing end-to-end evaluation, FinSearchComp offers a professional, high-difficulty testbed for complex financial search and reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_13160 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning Hu, Liang Jiao, Jianpeng Liu, Jiashuo Ren, Yanle Wen, Zhoufutu Zhang, Kaiyuan Zhang, Xuanliang Gao, Xiang He, Tianci Hu, Fei Liao, Yali Wang, Zaiyuan Yang, Chenghao Yang, Qianyu Yin, Mingren Zeng, Zhiyuan Zhang, Ge Zhang, Xinyi Zhao, Xiying Zhu, Zhenwei Namkoong, Hongseok Huang, Wenhao Tang, Yuwen Machine Learning Artificial Intelligence Search has emerged as core infrastructure for LLM-based agents and is widely viewed as critical on the path toward more general intelligence. Finance is a particularly demanding proving ground: analysts routinely conduct complex, multi-step searches over time-sensitive, domain-specific data, making it ideal for assessing both search proficiency and knowledge-grounded reasoning. Yet no existing open financial datasets evaluate data searching capability of end-to-end agents, largely because constructing realistic, complicated tasks requires deep financial expertise and time-sensitive data is hard to evaluate. We present FinSearchComp, the first fully open-source agent benchmark for realistic, open-domain financial search and reasoning. FinSearchComp comprises three tasks -- Time-Sensitive Data Fetching, Simple Historical Lookup, and Complex Historical Investigation -- closely reproduce real-world financial analyst workflows. To ensure difficulty and reliability, we engage 70 professional financial experts for annotation and implement a rigorous multi-stage quality-assurance pipeline. The benchmark includes 635 questions spanning global and Greater China markets, and we evaluate 21 models (products) on it. Grok 4 (web) tops the global subset, approaching expert-level accuracy. DouBao (web) leads on the Greater China subset. Experimental analyses show that equipping agents with web search and financial plugins substantially improves results on FinSearchComp, and the country origin of models and tools impact performance significantly.By aligning with realistic analyst tasks and providing end-to-end evaluation, FinSearchComp offers a professional, high-difficulty testbed for complex financial search and reasoning. |
| title | FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2509.13160 |