Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.13233 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913273411010560 |
|---|---|
| author | Zhao, Shuaijiang Fang, Xiaoquan |
| author_facet | Zhao, Shuaijiang Fang, Xiaoquan |
| contents | In the era of flourishing large-scale models, the challenge of selecting and optimizing datasets from the vast and complex sea of data, to enhance the performance of large language models within the constraints of limited computational resources, has become paramount. This paper details our solution for the BetterMixture challenge, which focuses on the fine-tuning data mixing for large language models. Our approach, which secured third place, incorporates data deduplication, low-level and high-level quality filtering, and diversity selection. The foundation of our solution is Ke-Data-Juicer, an extension of Data-Juicer, demonstrating its robust capabilities in handling and optimizing data for large language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_13233 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Technical Report: Competition Solution For BetterMixture Zhao, Shuaijiang Fang, Xiaoquan Computation and Language In the era of flourishing large-scale models, the challenge of selecting and optimizing datasets from the vast and complex sea of data, to enhance the performance of large language models within the constraints of limited computational resources, has become paramount. This paper details our solution for the BetterMixture challenge, which focuses on the fine-tuning data mixing for large language models. Our approach, which secured third place, incorporates data deduplication, low-level and high-level quality filtering, and diversity selection. The foundation of our solution is Ke-Data-Juicer, an extension of Data-Juicer, demonstrating its robust capabilities in handling and optimizing data for large language models. |
| title | Technical Report: Competition Solution For BetterMixture |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2403.13233 |