Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910836416577536 |
|---|---|
| author | Tang, Shuo Pang, Xianghe Liu, Zexi Tang, Bohan Ye, Rui Jin, Tian Dong, Xiaowen Wang, Yanfeng Chen, Siheng |
| author_facet | Tang, Shuo Pang, Xianghe Liu, Zexi Tang, Bohan Ye, Rui Jin, Tian Dong, Xiaowen Wang, Yanfeng Chen, Siheng |
| contents | Post-training is essential for enabling large language models (LLMs) to follow human instructions. However, its effectiveness depends on high-quality instruction data, which is challenging to obtain in the real world due to privacy concerns, data scarcity, and high annotation costs. To fill this gap, inspired by the recent success of using LLMs to simulate human society, we propose MATRIX, a multi-agent simulator that automatically generates diverse text-based scenarios, capturing a wide range of real-world human needs in a realistic and scalable manner. Leveraging these outputs, we introduce a novel scenario-driven instruction generator MATRIX-Gen for controllable and highly realistic data synthesis. Extensive experiments demonstrate that our framework effectively generates both general and domain-specific data. On AlpacaEval 2 and Arena-Hard benchmarks, Llama-3-8B-Base, post-trained on datasets synthesized by MATRIX-Gen with just 20K instruction-response pairs, outperforms Meta's Llama-3-8B-Instruct model, which was trained on over 10M pairs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_14251 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation Tang, Shuo Pang, Xianghe Liu, Zexi Tang, Bohan Ye, Rui Jin, Tian Dong, Xiaowen Wang, Yanfeng Chen, Siheng Artificial Intelligence Computation and Language Post-training is essential for enabling large language models (LLMs) to follow human instructions. However, its effectiveness depends on high-quality instruction data, which is challenging to obtain in the real world due to privacy concerns, data scarcity, and high annotation costs. To fill this gap, inspired by the recent success of using LLMs to simulate human society, we propose MATRIX, a multi-agent simulator that automatically generates diverse text-based scenarios, capturing a wide range of real-world human needs in a realistic and scalable manner. Leveraging these outputs, we introduce a novel scenario-driven instruction generator MATRIX-Gen for controllable and highly realistic data synthesis. Extensive experiments demonstrate that our framework effectively generates both general and domain-specific data. On AlpacaEval 2 and Arena-Hard benchmarks, Llama-3-8B-Base, post-trained on datasets synthesized by MATRIX-Gen with just 20K instruction-response pairs, outperforms Meta's Llama-3-8B-Instruct model, which was trained on over 10M pairs. |
| title | Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2410.14251 |