Focused Large Language Models are Stable Many-Shot Learners
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910576239706112 |
|---|---|
| author | Yuan, Peiwen Feng, Shaoxiong Li, Yiwei Wang, Xinglin Zhang, Yueqi Tan, Chuyi Pan, Boyuan Wang, Heda Hu, Yao Li, Kan |
| author_facet | Yuan, Peiwen Feng, Shaoxiong Li, Yiwei Wang, Xinglin Zhang, Yueqi Tan, Chuyi Pan, Boyuan Wang, Heda Hu, Yao Li, Kan |
| contents | In-Context Learning (ICL) enables large language models (LLMs) to achieve rapid task adaptation by learning from demonstrations. With the increase in available context length of LLMs, recent experiments have shown that the performance of ICL does not necessarily scale well in many-shot (demonstration) settings. We theoretically and experimentally confirm that the reason lies in more demonstrations dispersing the model attention from the query, hindering its understanding of key content. Inspired by how humans learn from examples, we propose a training-free method FocusICL, which conducts triviality filtering to avoid attention being diverted by unimportant contents at token-level and operates hierarchical attention to further ensure sufficient attention towards current query at demonstration-level. We also design an efficient hyperparameter searching strategy for FocusICL based on model perplexity of demonstrations. Comprehensive experiments validate that FocusICL achieves an average performance improvement of 5.2% over vanilla ICL and scales well with many-shot demonstrations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_13987 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Focused Large Language Models are Stable Many-Shot Learners Yuan, Peiwen Feng, Shaoxiong Li, Yiwei Wang, Xinglin Zhang, Yueqi Tan, Chuyi Pan, Boyuan Wang, Heda Hu, Yao Li, Kan Computation and Language Artificial Intelligence In-Context Learning (ICL) enables large language models (LLMs) to achieve rapid task adaptation by learning from demonstrations. With the increase in available context length of LLMs, recent experiments have shown that the performance of ICL does not necessarily scale well in many-shot (demonstration) settings. We theoretically and experimentally confirm that the reason lies in more demonstrations dispersing the model attention from the query, hindering its understanding of key content. Inspired by how humans learn from examples, we propose a training-free method FocusICL, which conducts triviality filtering to avoid attention being diverted by unimportant contents at token-level and operates hierarchical attention to further ensure sufficient attention towards current query at demonstration-level. We also design an efficient hyperparameter searching strategy for FocusICL based on model perplexity of demonstrations. Comprehensive experiments validate that FocusICL achieves an average performance improvement of 5.2% over vanilla ICL and scales well with many-shot demonstrations. |
| title | Focused Large Language Models are Stable Many-Shot Learners |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2408.13987 |