Instruction-Tuning Data Synthesis from Scratch via Web Reconstruction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908372688699392 |
|---|---|
| author | Jiang, Yuxin Wang, Yufei Wu, Chuhan Dai, Xinyi Xu, Yan Gan, Weinan Wang, Yasheng Jiang, Xin Shang, Lifeng Tang, Ruiming Wang, Wei |
| author_facet | Jiang, Yuxin Wang, Yufei Wu, Chuhan Dai, Xinyi Xu, Yan Gan, Weinan Wang, Yasheng Jiang, Xin Shang, Lifeng Tang, Ruiming Wang, Wei |
| contents | The improvement of LLMs' instruction-following capabilities depends critically on the availability of high-quality instruction-response pairs. While existing automatic data synthetic methods alleviate the burden of manual curation, they often rely heavily on either the quality of seed data or strong assumptions about the structure and content of web documents. To tackle these challenges, we propose Web Reconstruction (WebR), a fully automated framework for synthesizing high-quality instruction-tuning (IT) data directly from raw web documents with minimal assumptions. Leveraging the inherent diversity of raw web content, we conceptualize web reconstruction as an instruction-tuning data synthesis task via a novel dual-perspective paradigm--Web as Instruction and Web as Response--where each web document is designated as either an instruction or a response to trigger the reconstruction process. Comprehensive experiments show that datasets generated by WebR outperform state-of-the-art baselines by up to 16.65% across four instruction-following benchmarks. Notably, WebR demonstrates superior compatibility, data efficiency, and scalability, enabling enhanced domain adaptation with minimal effort. The data and code are publicly available at https://github.com/YJiangcm/WebR. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_15573 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Instruction-Tuning Data Synthesis from Scratch via Web Reconstruction Jiang, Yuxin Wang, Yufei Wu, Chuhan Dai, Xinyi Xu, Yan Gan, Weinan Wang, Yasheng Jiang, Xin Shang, Lifeng Tang, Ruiming Wang, Wei Computation and Language The improvement of LLMs' instruction-following capabilities depends critically on the availability of high-quality instruction-response pairs. While existing automatic data synthetic methods alleviate the burden of manual curation, they often rely heavily on either the quality of seed data or strong assumptions about the structure and content of web documents. To tackle these challenges, we propose Web Reconstruction (WebR), a fully automated framework for synthesizing high-quality instruction-tuning (IT) data directly from raw web documents with minimal assumptions. Leveraging the inherent diversity of raw web content, we conceptualize web reconstruction as an instruction-tuning data synthesis task via a novel dual-perspective paradigm--Web as Instruction and Web as Response--where each web document is designated as either an instruction or a response to trigger the reconstruction process. Comprehensive experiments show that datasets generated by WebR outperform state-of-the-art baselines by up to 16.65% across four instruction-following benchmarks. Notably, WebR demonstrates superior compatibility, data efficiency, and scalability, enabling enhanced domain adaptation with minimal effort. The data and code are publicly available at https://github.com/YJiangcm/WebR. |
| title | Instruction-Tuning Data Synthesis from Scratch via Web Reconstruction |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.15573 |