Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2403.12766 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909589176320000 |
|---|---|
| author | Wang, Cunxiang Ning, Ruoxi Pan, Boqi Wu, Tonghui Guo, Qipeng Deng, Cheng Bao, Guangsheng Hu, Xiangkun Zhang, Zheng Wang, Qian Zhang, Yue |
| author_facet | Wang, Cunxiang Ning, Ruoxi Pan, Boqi Wu, Tonghui Guo, Qipeng Deng, Cheng Bao, Guangsheng Hu, Xiangkun Zhang, Zheng Wang, Qian Zhang, Yue |
| contents | Recent advancements in Large Language Models (LLMs) have pushed the boundaries of natural language processing, especially in long-context understanding. However, the evaluation of these models' long-context abilities remains a challenge due to the limitations of current benchmarks. To address this gap, we introduce NovelQA, a benchmark tailored for evaluating LLMs with complex, extended narratives. Constructed from English novels, NovelQA offers a unique blend of complexity, length, and narrative coherence, making it an ideal tool for assessing deep textual understanding in LLMs. This paper details the design and construction of NovelQA, focusing on its comprehensive manual annotation process and the variety of question types aimed at evaluating nuanced comprehension. Our evaluation of long-context LLMs on NovelQA reveals significant insights into their strengths and weaknesses. Notably, the models struggle with multi-hop reasoning, detail-oriented questions, and handling extremely long inputs, with average lengths exceeding 200,000 tokens. Results highlight the need for substantial advancements in LLMs to enhance their long-context comprehension and contribute effectively to computational literary analysis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_12766 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens Wang, Cunxiang Ning, Ruoxi Pan, Boqi Wu, Tonghui Guo, Qipeng Deng, Cheng Bao, Guangsheng Hu, Xiangkun Zhang, Zheng Wang, Qian Zhang, Yue Computation and Language Recent advancements in Large Language Models (LLMs) have pushed the boundaries of natural language processing, especially in long-context understanding. However, the evaluation of these models' long-context abilities remains a challenge due to the limitations of current benchmarks. To address this gap, we introduce NovelQA, a benchmark tailored for evaluating LLMs with complex, extended narratives. Constructed from English novels, NovelQA offers a unique blend of complexity, length, and narrative coherence, making it an ideal tool for assessing deep textual understanding in LLMs. This paper details the design and construction of NovelQA, focusing on its comprehensive manual annotation process and the variety of question types aimed at evaluating nuanced comprehension. Our evaluation of long-context LLMs on NovelQA reveals significant insights into their strengths and weaknesses. Notably, the models struggle with multi-hop reasoning, detail-oriented questions, and handling extremely long inputs, with average lengths exceeding 200,000 tokens. Results highlight the need for substantial advancements in LLMs to enhance their long-context comprehension and contribute effectively to computational literary analysis. |
| title | NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2403.12766 |