FinDVer: Explainable Claim Verification over Long and Hybrid-Content Financial Documents
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916474122141696 |
|---|---|
| author | Zhao, Yilun Long, Yitao Jiang, Yuru Wang, Chengye Chen, Weiyuan Liu, Hongjun Zhang, Yiming Tang, Xiangru Zhao, Chen Cohan, Arman |
| author_facet | Zhao, Yilun Long, Yitao Jiang, Yuru Wang, Chengye Chen, Weiyuan Liu, Hongjun Zhang, Yiming Tang, Xiangru Zhao, Chen Cohan, Arman |
| contents | We introduce FinDVer, a comprehensive benchmark specifically designed to evaluate the explainable claim verification capabilities of LLMs in the context of understanding and analyzing long, hybrid-content financial documents. FinDVer contains 2,400 expert-annotated examples, divided into three subsets: information extraction, numerical reasoning, and knowledge-intensive reasoning, each addressing common scenarios encountered in real-world financial contexts. We assess a broad spectrum of LLMs under long-context and RAG settings. Our results show that even the current best-performing system, GPT-4o, still lags behind human experts. We further provide in-depth analysis on long-context and RAG setting, Chain-of-Thought reasoning, and model reasoning errors, offering insights to drive future advancements. We believe that FinDVer can serve as a valuable benchmark for evaluating LLMs in claim verification over complex, expert-domain documents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_05764 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FinDVer: Explainable Claim Verification over Long and Hybrid-Content Financial Documents Zhao, Yilun Long, Yitao Jiang, Yuru Wang, Chengye Chen, Weiyuan Liu, Hongjun Zhang, Yiming Tang, Xiangru Zhao, Chen Cohan, Arman Computation and Language Machine Learning We introduce FinDVer, a comprehensive benchmark specifically designed to evaluate the explainable claim verification capabilities of LLMs in the context of understanding and analyzing long, hybrid-content financial documents. FinDVer contains 2,400 expert-annotated examples, divided into three subsets: information extraction, numerical reasoning, and knowledge-intensive reasoning, each addressing common scenarios encountered in real-world financial contexts. We assess a broad spectrum of LLMs under long-context and RAG settings. Our results show that even the current best-performing system, GPT-4o, still lags behind human experts. We further provide in-depth analysis on long-context and RAG setting, Chain-of-Thought reasoning, and model reasoning errors, offering insights to drive future advancements. We believe that FinDVer can serve as a valuable benchmark for evaluating LLMs in claim verification over complex, expert-domain documents. |
| title | FinDVer: Explainable Claim Verification over Long and Hybrid-Content Financial Documents |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2411.05764 |