Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2310.19019 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929421559005184 |
|---|---|
| author | He, Nan Lai, Hanyu Zhao, Chenyang Cheng, Zirui Pan, Junting Qin, Ruoyu Lu, Ruofan Lu, Rui Zhang, Yunchen Zhao, Gangming Hou, Zhaohui Huang, Zhiyuan Lu, Shaoqing Liang, Ding Zhan, Mingjie |
| author_facet | He, Nan Lai, Hanyu Zhao, Chenyang Cheng, Zirui Pan, Junting Qin, Ruoyu Lu, Ruofan Lu, Rui Zhang, Yunchen Zhao, Gangming Hou, Zhaohui Huang, Zhiyuan Lu, Shaoqing Liang, Ding Zhan, Mingjie |
| contents | Large Language Models (LLMs) exhibit impressive reasoning and data augmentation capabilities in various NLP tasks. However, what about small models? In this work, we propose TeacherLM-7.1B, capable of annotating relevant fundamentals, chain of thought, and common mistakes for most NLP samples, which makes annotation more than just an answer, thus allowing other models to learn "why" instead of just "what". The TeacherLM-7.1B model achieved a zero-shot score of 52.3 on MMLU, surpassing most models with over 100B parameters. Even more remarkable is its data augmentation ability. Based on TeacherLM-7.1B, we augmented 58 NLP datasets and taught various student models with different parameters from OPT and BLOOM series in a multi-task setting. The experimental results indicate that the data augmentation provided by TeacherLM has brought significant benefits. We will release the TeacherLM series of models and augmented datasets as open-source. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_19019 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | TeacherLM: Teaching to Fish Rather Than Giving the Fish, Language Modeling Likewise He, Nan Lai, Hanyu Zhao, Chenyang Cheng, Zirui Pan, Junting Qin, Ruoyu Lu, Ruofan Lu, Rui Zhang, Yunchen Zhao, Gangming Hou, Zhaohui Huang, Zhiyuan Lu, Shaoqing Liang, Ding Zhan, Mingjie Computation and Language Artificial Intelligence Large Language Models (LLMs) exhibit impressive reasoning and data augmentation capabilities in various NLP tasks. However, what about small models? In this work, we propose TeacherLM-7.1B, capable of annotating relevant fundamentals, chain of thought, and common mistakes for most NLP samples, which makes annotation more than just an answer, thus allowing other models to learn "why" instead of just "what". The TeacherLM-7.1B model achieved a zero-shot score of 52.3 on MMLU, surpassing most models with over 100B parameters. Even more remarkable is its data augmentation ability. Based on TeacherLM-7.1B, we augmented 58 NLP datasets and taught various student models with different parameters from OPT and BLOOM series in a multi-task setting. The experimental results indicate that the data augmentation provided by TeacherLM has brought significant benefits. We will release the TeacherLM series of models and augmented datasets as open-source. |
| title | TeacherLM: Teaching to Fish Rather Than Giving the Fish, Language Modeling Likewise |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2310.19019 |