Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909333495742464 |
|---|---|
| author | Wang, Minzheng Chen, Longze Fu, Cheng Liao, Shengyi Zhang, Xinghua Wu, Bingli Yu, Haiyang Xu, Nan Zhang, Lei Luo, Run Li, Yunshui Yang, Min Huang, Fei Li, Yongbin |
| author_facet | Wang, Minzheng Chen, Longze Fu, Cheng Liao, Shengyi Zhang, Xinghua Wu, Bingli Yu, Haiyang Xu, Nan Zhang, Lei Luo, Run Li, Yunshui Yang, Min Huang, Fei Li, Yongbin |
| contents | Long-context modeling capabilities have garnered widespread attention, leading to the emergence of Large Language Models (LLMs) with ultra-context windows. Meanwhile, benchmarks for evaluating long-context LLMs are gradually catching up. However, existing benchmarks employ irrelevant noise texts to artificially extend the length of test cases, diverging from the real-world scenarios of long-context applications. To bridge this gap, we propose a novel long-context benchmark, Loong, aligning with realistic scenarios through extended multi-document question answering (QA). Unlike typical document QA, in Loong's test cases, each document is relevant to the final answer, ignoring any document will lead to the failure of the answer. Furthermore, Loong introduces four types of tasks with a range of context lengths: Spotlight Locating, Comparison, Clustering, and Chain of Reasoning, to facilitate a more realistic and comprehensive evaluation of long-context understanding. Extensive experiments indicate that existing long-context language models still exhibit considerable potential for enhancement. Retrieval augmented generation (RAG) achieves poor performance, demonstrating that Loong can reliably assess the model's long-context modeling capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_17419 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA Wang, Minzheng Chen, Longze Fu, Cheng Liao, Shengyi Zhang, Xinghua Wu, Bingli Yu, Haiyang Xu, Nan Zhang, Lei Luo, Run Li, Yunshui Yang, Min Huang, Fei Li, Yongbin Computation and Language Artificial Intelligence Long-context modeling capabilities have garnered widespread attention, leading to the emergence of Large Language Models (LLMs) with ultra-context windows. Meanwhile, benchmarks for evaluating long-context LLMs are gradually catching up. However, existing benchmarks employ irrelevant noise texts to artificially extend the length of test cases, diverging from the real-world scenarios of long-context applications. To bridge this gap, we propose a novel long-context benchmark, Loong, aligning with realistic scenarios through extended multi-document question answering (QA). Unlike typical document QA, in Loong's test cases, each document is relevant to the final answer, ignoring any document will lead to the failure of the answer. Furthermore, Loong introduces four types of tasks with a range of context lengths: Spotlight Locating, Comparison, Clustering, and Chain of Reasoning, to facilitate a more realistic and comprehensive evaluation of long-context understanding. Extensive experiments indicate that existing long-context language models still exhibit considerable potential for enhancement. Retrieval augmented generation (RAG) achieves poor performance, demonstrating that Loong can reliably assess the model's long-context modeling capabilities. |
| title | Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2406.17419 |