Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2406.10908 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909349019910144 |
|---|---|
| author | Zixiao, Zhu Zijian, Feng Hanzhang, Zhou Junlang, Qian Kezhi, Mao |
| author_facet | Zixiao, Zhu Zijian, Feng Hanzhang, Zhou Junlang, Qian Kezhi, Mao |
| contents | Effective organization of in-context learning (ICL) demonstrations is key to improving the quality of large language model (LLM) responses. To create better sample-label pairs that instruct LLM understanding, we introduce logit separability, a criterion to assess the clarity of both samples and class-related words at the logit level. This facilitates the optimization of sample and label selection, enhancing the precision of information provided in ICL demonstrations. Additionally, we find that incorporating multiple class-related words for each sample, rather than relying on a single class name, improves performance by offering a broader range of label information. Building on these insights, we propose LICL, a logit separability-based method that jointly organizes samples and integrates multiple class-related words into each sample-label pair. Evaluations across seven classification datasets show that this approach significantly improves ICL performance by providing clearer instructions and richer label information. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_10908 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Logit Separability-Driven Samples and Multiple Class-Related Words Selection for Advancing In-Context Learning Zixiao, Zhu Zijian, Feng Hanzhang, Zhou Junlang, Qian Kezhi, Mao Computation and Language Effective organization of in-context learning (ICL) demonstrations is key to improving the quality of large language model (LLM) responses. To create better sample-label pairs that instruct LLM understanding, we introduce logit separability, a criterion to assess the clarity of both samples and class-related words at the logit level. This facilitates the optimization of sample and label selection, enhancing the precision of information provided in ICL demonstrations. Additionally, we find that incorporating multiple class-related words for each sample, rather than relying on a single class name, improves performance by offering a broader range of label information. Building on these insights, we propose LICL, a logit separability-based method that jointly organizes samples and integrates multiple class-related words into each sample-label pair. Evaluations across seven classification datasets show that this approach significantly improves ICL performance by providing clearer instructions and richer label information. |
| title | Logit Separability-Driven Samples and Multiple Class-Related Words Selection for Advancing In-Context Learning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.10908 |