Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.16511 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915259523006464 |
|---|---|
| author | Liu, Fengze Zhou, Weidong Liu, Binbin Yu, Zhimiao Zhang, Yifan Lin, Haobin Yu, Yifeng Zhang, Bingni Zhou, Xiaohuan Wang, Taifeng Cao, Yong |
| author_facet | Liu, Fengze Zhou, Weidong Liu, Binbin Yu, Zhimiao Zhang, Yifan Lin, Haobin Yu, Yifeng Zhang, Bingni Zhou, Xiaohuan Wang, Taifeng Cao, Yong |
| contents | Quality and diversity are two critical metrics for the training data of large language models (LLMs), positively impacting performance. Existing studies often optimize these metrics separately, typically by first applying quality filtering and then adjusting data proportions. However, these approaches overlook the inherent trade-off between quality and diversity, necessitating their joint consideration. Given a fixed training quota, it is essential to evaluate both the quality of each data point and its complementary effect on the overall dataset. In this paper, we introduce a unified data selection framework called QuaDMix, which automatically optimizes the data distribution for LLM pretraining while balancing both quality and diversity. Specifically, we first propose multiple criteria to measure data quality and employ domain classification to distinguish data points, thereby measuring overall diversity. QuaDMix then employs a unified parameterized data sampling function that determines the sampling probability of each data point based on these quality and diversity related labels. To accelerate the search for the optimal parameters involved in the QuaDMix framework, we conduct simulated experiments on smaller models and use LightGBM for parameters searching, inspired by the RegMix method. Our experiments across diverse models and datasets demonstrate that QuaDMix achieves an average performance improvement of 7.2% across multiple benchmarks. These results outperform the independent strategies for quality and diversity, highlighting the necessity and ability to balance data quality and diversity. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_16511 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretraining Liu, Fengze Zhou, Weidong Liu, Binbin Yu, Zhimiao Zhang, Yifan Lin, Haobin Yu, Yifeng Zhang, Bingni Zhou, Xiaohuan Wang, Taifeng Cao, Yong Computation and Language Quality and diversity are two critical metrics for the training data of large language models (LLMs), positively impacting performance. Existing studies often optimize these metrics separately, typically by first applying quality filtering and then adjusting data proportions. However, these approaches overlook the inherent trade-off between quality and diversity, necessitating their joint consideration. Given a fixed training quota, it is essential to evaluate both the quality of each data point and its complementary effect on the overall dataset. In this paper, we introduce a unified data selection framework called QuaDMix, which automatically optimizes the data distribution for LLM pretraining while balancing both quality and diversity. Specifically, we first propose multiple criteria to measure data quality and employ domain classification to distinguish data points, thereby measuring overall diversity. QuaDMix then employs a unified parameterized data sampling function that determines the sampling probability of each data point based on these quality and diversity related labels. To accelerate the search for the optimal parameters involved in the QuaDMix framework, we conduct simulated experiments on smaller models and use LightGBM for parameters searching, inspired by the RegMix method. Our experiments across diverse models and datasets demonstrate that QuaDMix achieves an average performance improvement of 7.2% across multiple benchmarks. These results outperform the independent strategies for quality and diversity, highlighting the necessity and ability to balance data quality and diversity. |
| title | QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretraining |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.16511 |