ICLEval: Evaluating In-Context Learning Ability of Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917860486414336 |
|---|---|
| author | Chen, Wentong Lin, Yankai Zhou, ZhenHao Huang, HongYun Jia, Yantao Cao, Zhao Wen, Ji-Rong |
| author_facet | Chen, Wentong Lin, Yankai Zhou, ZhenHao Huang, HongYun Jia, Yantao Cao, Zhao Wen, Ji-Rong |
| contents | In-Context Learning (ICL) is a critical capability of Large Language Models (LLMs) as it empowers them to comprehend and reason across interconnected inputs. Evaluating the ICL ability of LLMs can enhance their utilization and deepen our understanding of how this ability is acquired at the training stage. However, existing evaluation frameworks primarily focus on language abilities and knowledge, often overlooking the assessment of ICL ability. In this work, we introduce the ICLEval benchmark to evaluate the ICL abilities of LLMs, which encompasses two key sub-abilities: exact copying and rule learning. Through the ICLEval benchmark, we demonstrate that ICL ability is universally present in different LLMs, and model size is not the sole determinant of ICL efficacy. Surprisingly, we observe that ICL abilities, particularly copying, develop early in the pretraining process and stabilize afterward. Our source codes and benchmark are released at https://github.com/yiye3/ICLEval. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_14955 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ICLEval: Evaluating In-Context Learning Ability of Large Language Models Chen, Wentong Lin, Yankai Zhou, ZhenHao Huang, HongYun Jia, Yantao Cao, Zhao Wen, Ji-Rong Computation and Language In-Context Learning (ICL) is a critical capability of Large Language Models (LLMs) as it empowers them to comprehend and reason across interconnected inputs. Evaluating the ICL ability of LLMs can enhance their utilization and deepen our understanding of how this ability is acquired at the training stage. However, existing evaluation frameworks primarily focus on language abilities and knowledge, often overlooking the assessment of ICL ability. In this work, we introduce the ICLEval benchmark to evaluate the ICL abilities of LLMs, which encompasses two key sub-abilities: exact copying and rule learning. Through the ICLEval benchmark, we demonstrate that ICL ability is universally present in different LLMs, and model size is not the sole determinant of ICL efficacy. Surprisingly, we observe that ICL abilities, particularly copying, develop early in the pretraining process and stabilize afterward. Our source codes and benchmark are released at https://github.com/yiye3/ICLEval. |
| title | ICLEval: Evaluating In-Context Learning Ability of Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.14955 |