100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912393203810304 |
|---|---|
| author | Yang, Wang Jin, Hongye Zhong, Shaochen Jiang, Song Wang, Qifan Chaudhary, Vipin Han, Xiaotian |
| author_facet | Yang, Wang Jin, Hongye Zhong, Shaochen Jiang, Song Wang, Qifan Chaudhary, Vipin Han, Xiaotian |
| contents | Long-context capability is considered one of the most important abilities of LLMs, as a truly long context-capable LLM enables users to effortlessly process many originally exhausting tasks -- e.g., digesting a long-form document to find answers vs. directly asking an LLM about it. However, existing real-task-based long-context evaluation benchmarks have two major shortcomings. First, benchmarks like LongBench often do not provide proper metrics to separate long-context performance from the model's baseline ability, making cross-model comparison unclear. Second, such benchmarks are usually constructed with fixed input lengths, which limits their applicability across different models and fails to reveal when a model begins to break down. To address these issues, we introduce a length-controllable long-context benchmark and a novel metric that disentangles baseline knowledge from true long-context capabilities. Experiments demonstrate the superiority of our approach in effectively evaluating LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_19293 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | 100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability? Yang, Wang Jin, Hongye Zhong, Shaochen Jiang, Song Wang, Qifan Chaudhary, Vipin Han, Xiaotian Computation and Language Artificial Intelligence Machine Learning Long-context capability is considered one of the most important abilities of LLMs, as a truly long context-capable LLM enables users to effortlessly process many originally exhausting tasks -- e.g., digesting a long-form document to find answers vs. directly asking an LLM about it. However, existing real-task-based long-context evaluation benchmarks have two major shortcomings. First, benchmarks like LongBench often do not provide proper metrics to separate long-context performance from the model's baseline ability, making cross-model comparison unclear. Second, such benchmarks are usually constructed with fixed input lengths, which limits their applicability across different models and fails to reveal when a model begins to break down. To address these issues, we introduce a length-controllable long-context benchmark and a novel metric that disentangles baseline knowledge from true long-context capabilities. Experiments demonstrate the superiority of our approach in effectively evaluating LLMs. |
| title | 100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability? |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2505.19293 |