DocFinQA: A Long-Context Financial Reasoning Dataset
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918170242056192 |
|---|---|
| author | Reddy, Varshini Koncel-Kedziorski, Rik Lai, Viet Dac Krumdick, Michael Lovering, Charles Tanner, Chris |
| author_facet | Reddy, Varshini Koncel-Kedziorski, Rik Lai, Viet Dac Krumdick, Michael Lovering, Charles Tanner, Chris |
| contents | For large language models (LLMs) to be effective in the financial domain -- where each decision can have a significant impact -- it is necessary to investigate realistic tasks and data. Financial professionals often interact with documents that are hundreds of pages long, but most financial research datasets only deal with short excerpts from these documents. To address this, we introduce a long-document financial QA task. We augment 7,437 questions from the existing FinQA dataset with the full-document context, extending the average context length from under 700 words in FinQA to 123k words in DocFinQA. We conduct extensive experiments over retrieval-based QA pipelines and long-context language models. DocFinQA proves a significant challenge for even state-of-the-art systems. We also provide a case-study on the longest documents in DocFinQA and find that models particularly struggle on these documents. Addressing these challenges may have a wide reaching impact across applications where specificity and long-range contexts are critical, like gene sequences and legal document contract analysis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_06915 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DocFinQA: A Long-Context Financial Reasoning Dataset Reddy, Varshini Koncel-Kedziorski, Rik Lai, Viet Dac Krumdick, Michael Lovering, Charles Tanner, Chris Computation and Language Artificial Intelligence For large language models (LLMs) to be effective in the financial domain -- where each decision can have a significant impact -- it is necessary to investigate realistic tasks and data. Financial professionals often interact with documents that are hundreds of pages long, but most financial research datasets only deal with short excerpts from these documents. To address this, we introduce a long-document financial QA task. We augment 7,437 questions from the existing FinQA dataset with the full-document context, extending the average context length from under 700 words in FinQA to 123k words in DocFinQA. We conduct extensive experiments over retrieval-based QA pipelines and long-context language models. DocFinQA proves a significant challenge for even state-of-the-art systems. We also provide a case-study on the longest documents in DocFinQA and find that models particularly struggle on these documents. Addressing these challenges may have a wide reaching impact across applications where specificity and long-range contexts are critical, like gene sequences and legal document contract analysis. |
| title | DocFinQA: A Long-Context Financial Reasoning Dataset |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2401.06915 |