Not All Documents Are What You Need for Extracting Instruction Tuning Data
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909615035252736 |
|---|---|
| author | Zhang, Chi Zhong, Huaping Li, Hongtao Chai, Chengliang Hong, Jiawei Deng, Yuhao Wang, Jiacheng Tan, Tian Yan, Yizhou Qiu, Jiantao Yuan, Ye Wang, Guoren He, Conghui Cao, Lei |
| author_facet | Zhang, Chi Zhong, Huaping Li, Hongtao Chai, Chengliang Hong, Jiawei Deng, Yuhao Wang, Jiacheng Tan, Tian Yan, Yizhou Qiu, Jiantao Yuan, Ye Wang, Guoren He, Conghui Cao, Lei |
| contents | Instruction tuning improves the performance of large language models (LLMs), but it heavily relies on high-quality training data. Recently, LLMs have been used to synthesize instruction data using seed question-answer (QA) pairs. However, these synthesized instructions often lack diversity and tend to be similar to the input seeds, limiting their applicability in real-world scenarios. To address this, we propose extracting instruction tuning data from web corpora that contain rich and diverse knowledge. A naive solution is to retrieve domain-specific documents and extract all QA pairs from them, but this faces two key challenges: (1) extracting all QA pairs using LLMs is prohibitively expensive, and (2) many extracted QA pairs may be irrelevant to the downstream tasks, potentially degrading model performance. To tackle these issues, we introduce EQUAL, an effective and scalable data extraction framework that iteratively alternates between document selection and high-quality QA pair extraction to enhance instruction tuning. EQUAL first clusters the document corpus based on embeddings derived from contrastive learning, then uses a multi-armed bandit strategy to efficiently identify clusters that are likely to contain valuable QA pairs. This iterative approach significantly reduces computational cost while boosting model performance. Experiments on AutoMathText and StackOverflow across four downstream tasks show that EQUAL reduces computational costs by 5-10x and improves accuracy by 2.5 percent on LLaMA-3.1-8B and Mistral-7B |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_12250 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Not All Documents Are What You Need for Extracting Instruction Tuning Data Zhang, Chi Zhong, Huaping Li, Hongtao Chai, Chengliang Hong, Jiawei Deng, Yuhao Wang, Jiacheng Tan, Tian Yan, Yizhou Qiu, Jiantao Yuan, Ye Wang, Guoren He, Conghui Cao, Lei Computation and Language Artificial Intelligence Instruction tuning improves the performance of large language models (LLMs), but it heavily relies on high-quality training data. Recently, LLMs have been used to synthesize instruction data using seed question-answer (QA) pairs. However, these synthesized instructions often lack diversity and tend to be similar to the input seeds, limiting their applicability in real-world scenarios. To address this, we propose extracting instruction tuning data from web corpora that contain rich and diverse knowledge. A naive solution is to retrieve domain-specific documents and extract all QA pairs from them, but this faces two key challenges: (1) extracting all QA pairs using LLMs is prohibitively expensive, and (2) many extracted QA pairs may be irrelevant to the downstream tasks, potentially degrading model performance. To tackle these issues, we introduce EQUAL, an effective and scalable data extraction framework that iteratively alternates between document selection and high-quality QA pair extraction to enhance instruction tuning. EQUAL first clusters the document corpus based on embeddings derived from contrastive learning, then uses a multi-armed bandit strategy to efficiently identify clusters that are likely to contain valuable QA pairs. This iterative approach significantly reduces computational cost while boosting model performance. Experiments on AutoMathText and StackOverflow across four downstream tasks show that EQUAL reduces computational costs by 5-10x and improves accuracy by 2.5 percent on LLaMA-3.1-8B and Mistral-7B |
| title | Not All Documents Are What You Need for Extracting Instruction Tuning Data |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2505.12250 |