SuperValid: Capability-Aligned OOD Validation for Generalizable Downstream Scaling
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910264830459904 |
|---|---|
| author | Sun, Quanen Tian, Changxin Shi, Ke Chen, Cai Peng, Cunyin Liu, Jia Chen, Kunlong Zhang, Zhiqiang |
| author_facet | Sun, Quanen Tian, Changxin Shi, Ke Chen, Cai Peng, Cunyin Liu, Jia Chen, Kunlong Zhang, Zhiqiang |
| contents | Scaling laws guide large language model training by relating compute to cross-entropy loss, and recent work further extends them to predict downstream benchmark performance. However, prior approaches face generalization limitations from two aspects: focusing on benchmark-level performance introduces scenario-specific artifacts, while relying on IID validation loss fails to track capability improvements when training distributions vary. In this work, we argue that downstream scaling should be studied at the capability level, which captures shared skill factors across related tasks while abstracting away benchmark-specific noise. We propose SuperValid, a framework that synthesizes OOD (out-of-distribution), capability-aligned validation data by distilling core concepts from benchmarks within a capability domain and expanding them into diverse, knowledge-rich texts. Extensive experiments spanning 17 benchmarks grouped into 6 capability domains show that SuperValid loss exhibits strong and stable correlation with downstream performance across models of different architectures, scales, and training data distributions. As a training-free metric computable during training without benchmark evaluation, SuperValid enables effective model selection, early stopping, and scaling decisions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_28179 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SuperValid: Capability-Aligned OOD Validation for Generalizable Downstream Scaling Sun, Quanen Tian, Changxin Shi, Ke Chen, Cai Peng, Cunyin Liu, Jia Chen, Kunlong Zhang, Zhiqiang Computation and Language Scaling laws guide large language model training by relating compute to cross-entropy loss, and recent work further extends them to predict downstream benchmark performance. However, prior approaches face generalization limitations from two aspects: focusing on benchmark-level performance introduces scenario-specific artifacts, while relying on IID validation loss fails to track capability improvements when training distributions vary. In this work, we argue that downstream scaling should be studied at the capability level, which captures shared skill factors across related tasks while abstracting away benchmark-specific noise. We propose SuperValid, a framework that synthesizes OOD (out-of-distribution), capability-aligned validation data by distilling core concepts from benchmarks within a capability domain and expanding them into diverse, knowledge-rich texts. Extensive experiments spanning 17 benchmarks grouped into 6 capability domains show that SuperValid loss exhibits strong and stable correlation with downstream performance across models of different architectures, scales, and training data distributions. As a training-free metric computable during training without benchmark evaluation, SuperValid enables effective model selection, early stopping, and scaling decisions. |
| title | SuperValid: Capability-Aligned OOD Validation for Generalizable Downstream Scaling |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.28179 |