Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909113202507776 |
|---|---|
| author | Li, Haoran Dong, Qingxiu Tang, Zhengyang Wang, Chaojun Zhang, Xingxing Huang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Zhang, Dongdong Gu, Yuxian Cheng, Xin Wang, Xun Chen, Si-Qing Dong, Li Lu, Wei Sui, Zhifang Wang, Benyou Lam, Wai Wei, Furu |
| author_facet | Li, Haoran Dong, Qingxiu Tang, Zhengyang Wang, Chaojun Zhang, Xingxing Huang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Zhang, Dongdong Gu, Yuxian Cheng, Xin Wang, Xun Chen, Si-Qing Dong, Li Lu, Wei Sui, Zhifang Wang, Benyou Lam, Wai Wei, Furu |
| contents | We introduce Generalized Instruction Tuning (called GLAN), a general and scalable method for instruction tuning of Large Language Models (LLMs). Unlike prior work that relies on seed examples or existing datasets to construct instruction tuning data, GLAN exclusively utilizes a pre-curated taxonomy of human knowledge and capabilities as input and generates large-scale synthetic instruction data across all disciplines. Specifically, inspired by the systematic structure in human education system, we build the taxonomy by decomposing human knowledge and capabilities to various fields, sub-fields and ultimately, distinct disciplines semi-automatically, facilitated by LLMs. Subsequently, we generate a comprehensive list of subjects for every discipline and proceed to design a syllabus tailored to each subject, again utilizing LLMs. With the fine-grained key concepts detailed in every class session of the syllabus, we are able to generate diverse instructions with a broad coverage across the entire spectrum of human knowledge and skills. Extensive experiments on large language models (e.g., Mistral) demonstrate that GLAN excels in multiple dimensions from mathematical reasoning, coding, academic exams, logical reasoning to general instruction following without using task-specific training data of these tasks. In addition, GLAN allows for easy customization and new fields or skills can be added by simply incorporating a new node into our taxonomy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_13064 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models Li, Haoran Dong, Qingxiu Tang, Zhengyang Wang, Chaojun Zhang, Xingxing Huang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Zhang, Dongdong Gu, Yuxian Cheng, Xin Wang, Xun Chen, Si-Qing Dong, Li Lu, Wei Sui, Zhifang Wang, Benyou Lam, Wai Wei, Furu Computation and Language We introduce Generalized Instruction Tuning (called GLAN), a general and scalable method for instruction tuning of Large Language Models (LLMs). Unlike prior work that relies on seed examples or existing datasets to construct instruction tuning data, GLAN exclusively utilizes a pre-curated taxonomy of human knowledge and capabilities as input and generates large-scale synthetic instruction data across all disciplines. Specifically, inspired by the systematic structure in human education system, we build the taxonomy by decomposing human knowledge and capabilities to various fields, sub-fields and ultimately, distinct disciplines semi-automatically, facilitated by LLMs. Subsequently, we generate a comprehensive list of subjects for every discipline and proceed to design a syllabus tailored to each subject, again utilizing LLMs. With the fine-grained key concepts detailed in every class session of the syllabus, we are able to generate diverse instructions with a broad coverage across the entire spectrum of human knowledge and skills. Extensive experiments on large language models (e.g., Mistral) demonstrate that GLAN excels in multiple dimensions from mathematical reasoning, coding, academic exams, logical reasoning to general instruction following without using task-specific training data of these tasks. In addition, GLAN allows for easy customization and new fields or skills can be added by simply incorporating a new node into our taxonomy. |
| title | Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.13064 |