UCFE: A User-Centric Financial Expertise Benchmark for Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917915933016064 |
|---|---|
| author | Yang, Yuzhe Zhang, Yifei Hu, Yan Guo, Yilin Gan, Ruoli He, Yueru Lei, Mingcong Zhang, Xiao Wang, Haining Xie, Qianqian Huang, Jimin Yu, Honghai Wang, Benyou |
| author_facet | Yang, Yuzhe Zhang, Yifei Hu, Yan Guo, Yilin Gan, Ruoli He, Yueru Lei, Mingcong Zhang, Xiao Wang, Haining Xie, Qianqian Huang, Jimin Yu, Honghai Wang, Benyou |
| contents | This paper introduces the UCFE: User-Centric Financial Expertise benchmark, an innovative framework designed to evaluate the ability of large language models (LLMs) to handle complex real-world financial tasks. UCFE benchmark adopts a hybrid approach that combines human expert evaluations with dynamic, task-specific interactions to simulate the complexities of evolving financial scenarios. Firstly, we conducted a user study involving 804 participants, collecting their feedback on financial tasks. Secondly, based on this feedback, we created our dataset that encompasses a wide range of user intents and interactions. This dataset serves as the foundation for benchmarking 11 LLMs services using the LLM-as-Judge methodology. Our results show a significant alignment between benchmark scores and human preferences, with a Pearson correlation coefficient of 0.78, confirming the effectiveness of the UCFE dataset and our evaluation approach. UCFE benchmark not only reveals the potential of LLMs in the financial domain but also provides a robust framework for assessing their performance and user satisfaction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_14059 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | UCFE: A User-Centric Financial Expertise Benchmark for Large Language Models Yang, Yuzhe Zhang, Yifei Hu, Yan Guo, Yilin Gan, Ruoli He, Yueru Lei, Mingcong Zhang, Xiao Wang, Haining Xie, Qianqian Huang, Jimin Yu, Honghai Wang, Benyou Computational Finance Computational Engineering, Finance, and Science Computation and Language This paper introduces the UCFE: User-Centric Financial Expertise benchmark, an innovative framework designed to evaluate the ability of large language models (LLMs) to handle complex real-world financial tasks. UCFE benchmark adopts a hybrid approach that combines human expert evaluations with dynamic, task-specific interactions to simulate the complexities of evolving financial scenarios. Firstly, we conducted a user study involving 804 participants, collecting their feedback on financial tasks. Secondly, based on this feedback, we created our dataset that encompasses a wide range of user intents and interactions. This dataset serves as the foundation for benchmarking 11 LLMs services using the LLM-as-Judge methodology. Our results show a significant alignment between benchmark scores and human preferences, with a Pearson correlation coefficient of 0.78, confirming the effectiveness of the UCFE dataset and our evaluation approach. UCFE benchmark not only reveals the potential of LLMs in the financial domain but also provides a robust framework for assessing their performance and user satisfaction. |
| title | UCFE: A User-Centric Financial Expertise Benchmark for Large Language Models |
| topic | Computational Finance Computational Engineering, Finance, and Science Computation and Language |
| url | https://arxiv.org/abs/2410.14059 |