Tulu 3: Pushing Frontiers in Open Language Model Post-Training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909579697192960 |
|---|---|
| author | Lambert, Nathan Morrison, Jacob Pyatkin, Valentina Huang, Shengyi Ivison, Hamish Brahman, Faeze Miranda, Lester James V. Liu, Alisa Dziri, Nouha Lyu, Shane Gu, Yuling Malik, Saumya Graf, Victoria Hwang, Jena D. Yang, Jiangjiang Bras, Ronan Le Tafjord, Oyvind Wilhelm, Chris Soldaini, Luca Smith, Noah A. Wang, Yizhong Dasigi, Pradeep Hajishirzi, Hannaneh |
| author_facet | Lambert, Nathan Morrison, Jacob Pyatkin, Valentina Huang, Shengyi Ivison, Hamish Brahman, Faeze Miranda, Lester James V. Liu, Alisa Dziri, Nouha Lyu, Shane Gu, Yuling Malik, Saumya Graf, Victoria Hwang, Jena D. Yang, Jiangjiang Bras, Ronan Le Tafjord, Oyvind Wilhelm, Chris Soldaini, Luca Smith, Noah A. Wang, Yizhong Dasigi, Pradeep Hajishirzi, Hannaneh |
| contents | Language model post-training is applied to refine behaviors and unlock new skills across a wide range of recent language models, but open recipes for applying these techniques lag behind proprietary ones. The underlying training data and recipes for post-training are simultaneously the most important pieces of the puzzle and the portion with the least transparency. To bridge this gap, we introduce Tulu 3, a family of fully-open state-of-the-art post-trained models, alongside its data, code, and training recipes, serving as a comprehensive guide for modern post-training techniques. Tulu 3, which builds on Llama 3.1 base models, achieves results surpassing the instruct versions of Llama 3.1, Qwen 2.5, Mistral, and even closed models such as GPT-4o-mini and Claude 3.5-Haiku. The training algorithms for our models include supervised finetuning (SFT), Direct Preference Optimization (DPO), and a novel method we call Reinforcement Learning with Verifiable Rewards (RLVR). With Tulu 3, we introduce a multi-task evaluation scheme for post-training recipes with development and unseen evaluations, standard benchmark implementations, and substantial decontamination of existing open datasets on said benchmarks. We conclude with analysis and discussion of training methods that did not reliably improve performance.
In addition to the Tulu 3 model weights and demo, we release the complete recipe -- including datasets for diverse core skills, a robust toolkit for data curation and evaluation, the training code and infrastructure, and, most importantly, a detailed report for reproducing and further adapting the Tulu 3 approach to more domains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_15124 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Tulu 3: Pushing Frontiers in Open Language Model Post-Training Lambert, Nathan Morrison, Jacob Pyatkin, Valentina Huang, Shengyi Ivison, Hamish Brahman, Faeze Miranda, Lester James V. Liu, Alisa Dziri, Nouha Lyu, Shane Gu, Yuling Malik, Saumya Graf, Victoria Hwang, Jena D. Yang, Jiangjiang Bras, Ronan Le Tafjord, Oyvind Wilhelm, Chris Soldaini, Luca Smith, Noah A. Wang, Yizhong Dasigi, Pradeep Hajishirzi, Hannaneh Computation and Language Language model post-training is applied to refine behaviors and unlock new skills across a wide range of recent language models, but open recipes for applying these techniques lag behind proprietary ones. The underlying training data and recipes for post-training are simultaneously the most important pieces of the puzzle and the portion with the least transparency. To bridge this gap, we introduce Tulu 3, a family of fully-open state-of-the-art post-trained models, alongside its data, code, and training recipes, serving as a comprehensive guide for modern post-training techniques. Tulu 3, which builds on Llama 3.1 base models, achieves results surpassing the instruct versions of Llama 3.1, Qwen 2.5, Mistral, and even closed models such as GPT-4o-mini and Claude 3.5-Haiku. The training algorithms for our models include supervised finetuning (SFT), Direct Preference Optimization (DPO), and a novel method we call Reinforcement Learning with Verifiable Rewards (RLVR). With Tulu 3, we introduce a multi-task evaluation scheme for post-training recipes with development and unseen evaluations, standard benchmark implementations, and substantial decontamination of existing open datasets on said benchmarks. We conclude with analysis and discussion of training methods that did not reliably improve performance. In addition to the Tulu 3 model weights and demo, we release the complete recipe -- including datasets for diverse core skills, a robust toolkit for data curation and evaluation, the training code and infrastructure, and, most importantly, a detailed report for reproducing and further adapting the Tulu 3 approach to more domains. |
| title | Tulu 3: Pushing Frontiers in Open Language Model Post-Training |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2411.15124 |