ATLAS: All-round Testing of Long-context Abilities across Scales
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914607196536832 |
|---|---|
| author | Huang, Deli Wang, Cunguang Tang, Hongyin Tang, Zhe Guo, Linsen Ru, Dongyu Yuan, Ruoshi Zhu, Ziyue Li, Xiaoyu Wang, Ziwen Zhang, Chen Gui, Anchun Zan, Wen Zhang, Jiaqi Cao, Xuezhi Wang, Jingang Cai, Xunliang Cao, Yixin |
| author_facet | Huang, Deli Wang, Cunguang Tang, Hongyin Tang, Zhe Guo, Linsen Ru, Dongyu Yuan, Ruoshi Zhu, Ziyue Li, Xiaoyu Wang, Ziwen Zhang, Chen Gui, Anchun Zan, Wen Zhang, Jiaqi Cao, Xuezhi Wang, Jingang Cai, Xunliang Cao, Yixin |
| contents | Long-context language models now advertise context windows up to millions of tokens, yet evaluations typically report a single length or a narrow task family, masking two failure modes: performance can collapse as length grows, and strong retrieval need not transfer to downstream use. We present ATLAS, a benchmarking framework that redefines long-context evaluation as length-dependent capability profiling. ATLAS contributes three methodological principles:(i) a layered taxonomy separating foundational operations from application workloads so failures can be attributed, (ii) length-aware AUC scoring that integrates score-length curves over a fixed 8K-1M grid, replacing single-point metrics with full degradation profiles, and (iii) ATLAScore, a harmonic-mean aggregate over taxonomy categories that penalizes imbalanced profiles, with end-to-end uncertainty propagation from subset scores through the nonlinear final aggregate. We instantiate the framework across eight capability dimensions with nine auditable components and 6,438 instances, and evaluate 26 models. Gemini-3.1-Pro-Preview leads at 128K, Claude-Opus-4.6 leads at 1M. Rankings reshuffle substantially between ATLASscore@8K-128K and ATLASscore@8K-1M: 7 models move by at least two ranks, and the two taxonomy layers share only 61% of cross-model variance, with individual rank gaps up to 12 positions. These results support reporting long-context quality by capability and length, not by a single headline score. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_28079 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ATLAS: All-round Testing of Long-context Abilities across Scales Huang, Deli Wang, Cunguang Tang, Hongyin Tang, Zhe Guo, Linsen Ru, Dongyu Yuan, Ruoshi Zhu, Ziyue Li, Xiaoyu Wang, Ziwen Zhang, Chen Gui, Anchun Zan, Wen Zhang, Jiaqi Cao, Xuezhi Wang, Jingang Cai, Xunliang Cao, Yixin Computation and Language Long-context language models now advertise context windows up to millions of tokens, yet evaluations typically report a single length or a narrow task family, masking two failure modes: performance can collapse as length grows, and strong retrieval need not transfer to downstream use. We present ATLAS, a benchmarking framework that redefines long-context evaluation as length-dependent capability profiling. ATLAS contributes three methodological principles:(i) a layered taxonomy separating foundational operations from application workloads so failures can be attributed, (ii) length-aware AUC scoring that integrates score-length curves over a fixed 8K-1M grid, replacing single-point metrics with full degradation profiles, and (iii) ATLAScore, a harmonic-mean aggregate over taxonomy categories that penalizes imbalanced profiles, with end-to-end uncertainty propagation from subset scores through the nonlinear final aggregate. We instantiate the framework across eight capability dimensions with nine auditable components and 6,438 instances, and evaluate 26 models. Gemini-3.1-Pro-Preview leads at 128K, Claude-Opus-4.6 leads at 1M. Rankings reshuffle substantially between ATLASscore@8K-128K and ATLASscore@8K-1M: 7 models move by at least two ranks, and the two taxonomy layers share only 61% of cross-model variance, with individual rank gaps up to 12 positions. These results support reporting long-context quality by capability and length, not by a single headline score. |
| title | ATLAS: All-round Testing of Long-context Abilities across Scales |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.28079 |