Capability Salience Vector: Fine-grained Alignment of Loss and Capabilities for Downstream Task Scaling Law
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908408728256512 |
|---|---|
| author | Ge, Qiming Xing, Shuhao Gao, Songyang Zhou, Yunhua Zou, Yicheng Zhang, Songyang Chen, Zhi Yan, Hang Zhang, Qi Guo, Qipeng Chen, Kai |
| author_facet | Ge, Qiming Xing, Shuhao Gao, Songyang Zhou, Yunhua Zou, Yicheng Zhang, Songyang Chen, Zhi Yan, Hang Zhang, Qi Guo, Qipeng Chen, Kai |
| contents | Scaling law builds the relationship between training computation and validation loss, enabling researchers to effectively predict the loss trending of models across different levels of computation. However, a gap still remains between validation loss and the model's downstream capabilities, making it untrivial to apply scaling law to direct performance prediction for downstream tasks. The loss typically represents a cumulative penalty for predicted tokens, which are implicitly considered to have equal importance. Nevertheless, our studies have shown evidence that when considering different training data distributions, we cannot directly model the relationship between downstream capability and computation or token loss. To bridge the gap between validation loss and downstream task capabilities, in this work, we introduce Capability Salience Vector, which decomposes the overall loss and assigns different importance weights to tokens to assess a specific meta-capability, aligning the validation loss with downstream task performance in terms of the model's capabilities. Experiments on various popular benchmarks demonstrate that our proposed Capability Salience Vector could significantly improve the predictability of language model performance on downstream tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_13216 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Capability Salience Vector: Fine-grained Alignment of Loss and Capabilities for Downstream Task Scaling Law Ge, Qiming Xing, Shuhao Gao, Songyang Zhou, Yunhua Zou, Yicheng Zhang, Songyang Chen, Zhi Yan, Hang Zhang, Qi Guo, Qipeng Chen, Kai Computation and Language Scaling law builds the relationship between training computation and validation loss, enabling researchers to effectively predict the loss trending of models across different levels of computation. However, a gap still remains between validation loss and the model's downstream capabilities, making it untrivial to apply scaling law to direct performance prediction for downstream tasks. The loss typically represents a cumulative penalty for predicted tokens, which are implicitly considered to have equal importance. Nevertheless, our studies have shown evidence that when considering different training data distributions, we cannot directly model the relationship between downstream capability and computation or token loss. To bridge the gap between validation loss and downstream task capabilities, in this work, we introduce Capability Salience Vector, which decomposes the overall loss and assigns different importance weights to tokens to assess a specific meta-capability, aligning the validation loss with downstream task performance in terms of the model's capabilities. Experiments on various popular benchmarks demonstrate that our proposed Capability Salience Vector could significantly improve the predictability of language model performance on downstream tasks. |
| title | Capability Salience Vector: Fine-grained Alignment of Loss and Capabilities for Downstream Task Scaling Law |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.13216 |