Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.17312 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929515280728064 |
|---|---|
| author | Tastet, Jean-Loup Timiryasov, Inar |
| author_facet | Tastet, Jean-Loup Timiryasov, Inar |
| contents | We present BabyLlama-2, a 345 million parameter model distillation-pretrained from two teachers on a 10 million word corpus for the BabyLM competition. On BLiMP and SuperGLUE benchmarks, BabyLlama-2 outperforms baselines trained on both 10 and 100 million word datasets with the same data mix, as well as its teacher models. Through an extensive hyperparameter sweep, we demonstrate that the advantages of distillation cannot be attributed to suboptimal hyperparameter selection of the teachers. Our findings underscore the need for further investigation into distillation techniques, particularly in data-limited settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_17312 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | BabyLlama-2: Ensemble-Distilled Models Consistently Outperform Teachers With Limited Data Tastet, Jean-Loup Timiryasov, Inar Computation and Language Machine Learning I.2.7 We present BabyLlama-2, a 345 million parameter model distillation-pretrained from two teachers on a 10 million word corpus for the BabyLM competition. On BLiMP and SuperGLUE benchmarks, BabyLlama-2 outperforms baselines trained on both 10 and 100 million word datasets with the same data mix, as well as its teacher models. Through an extensive hyperparameter sweep, we demonstrate that the advantages of distillation cannot be attributed to suboptimal hyperparameter selection of the teachers. Our findings underscore the need for further investigation into distillation techniques, particularly in data-limited settings. |
| title | BabyLlama-2: Ensemble-Distilled Models Consistently Outperform Teachers With Limited Data |
| topic | Computation and Language Machine Learning I.2.7 |
| url | https://arxiv.org/abs/2409.17312 |