LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916829858889728 |
|---|---|
| author | Tang, Zecheng Wang, Haitian Qiu, Quantong Ji, Baibei Sun, Ruoxi Zhou, Keyan Li, Juntao Zhang, Min |
| author_facet | Tang, Zecheng Wang, Haitian Qiu, Quantong Ji, Baibei Sun, Ruoxi Zhou, Keyan Li, Juntao Zhang, Min |
| contents | Long-context processing has become a fundamental capability for large language models~(LLMs). To assess model's long-context performance, numerous long-context evaluation benchmarks have been proposed. However, variations in evaluation settings across these benchmarks lead to inconsistent results, making it difficult to draw reliable comparisons. Besides, the high computational cost of long-context evaluation poses a significant barrier for the community to conduct comprehensive assessments of long-context models. In this paper, we propose LOOM-Scope, a comprehensive and efficient framework for long-context evaluation. LOOM-Scope standardizes evaluation settings across diverse benchmarks, supports deployment of efficient long-context inference acceleration methods, and introduces a holistic yet lightweight benchmark suite to evaluate models comprehensively. Homepage: https://loomscope.github.io |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_04723 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework Tang, Zecheng Wang, Haitian Qiu, Quantong Ji, Baibei Sun, Ruoxi Zhou, Keyan Li, Juntao Zhang, Min Computation and Language Long-context processing has become a fundamental capability for large language models~(LLMs). To assess model's long-context performance, numerous long-context evaluation benchmarks have been proposed. However, variations in evaluation settings across these benchmarks lead to inconsistent results, making it difficult to draw reliable comparisons. Besides, the high computational cost of long-context evaluation poses a significant barrier for the community to conduct comprehensive assessments of long-context models. In this paper, we propose LOOM-Scope, a comprehensive and efficient framework for long-context evaluation. LOOM-Scope standardizes evaluation settings across diverse benchmarks, supports deployment of efficient long-context inference acceleration methods, and introduces a holistic yet lightweight benchmark suite to evaluate models comprehensively. Homepage: https://loomscope.github.io |
| title | LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2507.04723 |