Predictable Emergent Abilities of LLMs: Proxy Tasks Are All You Need
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915055840264192 |
|---|---|
| author | Zhang, Bo-Wen Yan, Yan Yang, Boxiang Xue, Yifei Liu, Guang |
| author_facet | Zhang, Bo-Wen Yan, Yan Yang, Boxiang Xue, Yifei Liu, Guang |
| contents | While scaling laws optimize training configurations for large language models (LLMs) through experiments on smaller or early-stage models, they fail to predict emergent abilities due to the absence of such capabilities in these models. To address this, we propose a method that predicts emergent abilities by leveraging proxy tasks. We begin by establishing relevance metrics between the target task and candidate tasks based on performance differences across multiple models. These candidate tasks are then validated for robustness with small model ensembles, leading to the selection of the most appropriate proxy tasks. The predicted performance on the target task is then derived by integrating the evaluation results of these proxies. In a case study on tool utilization capabilities, our method demonstrated a strong correlation between predicted and actual performance, confirming its effectiveness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_07111 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Predictable Emergent Abilities of LLMs: Proxy Tasks Are All You Need Zhang, Bo-Wen Yan, Yan Yang, Boxiang Xue, Yifei Liu, Guang Computation and Language While scaling laws optimize training configurations for large language models (LLMs) through experiments on smaller or early-stage models, they fail to predict emergent abilities due to the absence of such capabilities in these models. To address this, we propose a method that predicts emergent abilities by leveraging proxy tasks. We begin by establishing relevance metrics between the target task and candidate tasks based on performance differences across multiple models. These candidate tasks are then validated for robustness with small model ensembles, leading to the selection of the most appropriate proxy tasks. The predicted performance on the target task is then derived by integrating the evaluation results of these proxies. In a case study on tool utilization capabilities, our method demonstrated a strong correlation between predicted and actual performance, confirming its effectiveness. |
| title | Predictable Emergent Abilities of LLMs: Proxy Tasks Are All You Need |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.07111 |