Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.10348 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929541026414592 |
|---|---|
| author | Shtok, Joseph Alfassy, Amit Dahood, Foad Abo Schwartz, Eliyahu Doveh, Sivan Arbelle, Assaf |
| author_facet | Shtok, Joseph Alfassy, Amit Dahood, Foad Abo Schwartz, Eliyahu Doveh, Sivan Arbelle, Assaf |
| contents | It has been shown that Large Language Models' (LLMs) performance can be improved for many tasks using Chain of Thought (CoT) or In-Context Learning (ICL), which involve demonstrating the steps needed to solve a task using a few examples. However, while datasets with input-output pairs are relatively easy to produce, providing demonstrations which include intermediate steps requires cumbersome manual work. These steps may be executable programs, as in agentic flows, or step-by-step reasoning as in CoT. In this work, we propose Automatic Data Labeling and Refinement (ADLR), a method to automatically generate and filter demonstrations which include the above intermediate steps, starting from a small seed of manually crafted examples. We demonstrate the advantage of ADLR in code-based table QA and mathematical reasoning, achieving up to a 5.5% gain. The code implementing our method is provided in the Supplementary material and will be made available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_10348 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Augmenting In-Context-Learning in LLMs via Automatic Data Labeling and Refinement Shtok, Joseph Alfassy, Amit Dahood, Foad Abo Schwartz, Eliyahu Doveh, Sivan Arbelle, Assaf Computation and Language It has been shown that Large Language Models' (LLMs) performance can be improved for many tasks using Chain of Thought (CoT) or In-Context Learning (ICL), which involve demonstrating the steps needed to solve a task using a few examples. However, while datasets with input-output pairs are relatively easy to produce, providing demonstrations which include intermediate steps requires cumbersome manual work. These steps may be executable programs, as in agentic flows, or step-by-step reasoning as in CoT. In this work, we propose Automatic Data Labeling and Refinement (ADLR), a method to automatically generate and filter demonstrations which include the above intermediate steps, starting from a small seed of manually crafted examples. We demonstrate the advantage of ADLR in code-based table QA and mathematical reasoning, achieving up to a 5.5% gain. The code implementing our method is provided in the Supplementary material and will be made available. |
| title | Augmenting In-Context-Learning in LLMs via Automatic Data Labeling and Refinement |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.10348 |