FIRE: Flexible Integration of Data Quality Ratings for Effective Pre-Training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912386234974208 |
|---|---|
| author | Xu, Liangyu Zhang, Xuemiao Duan, Feiyu Wang, Sirui Weng, Rongxiang Wang, Jingang Cai, Xunliang |
| author_facet | Xu, Liangyu Zhang, Xuemiao Duan, Feiyu Wang, Sirui Weng, Rongxiang Wang, Jingang Cai, Xunliang |
| contents | Selecting high-quality data can improve the pretraining efficiency of large language models (LLMs). Existing methods generally rely on heuristic techniques or single quality signals, limiting their ability to evaluate data quality comprehensively. In this work, we propose FIRE, a flexible and scalable framework for integrating multiple data quality raters, which allows for a comprehensive assessment of data quality across various dimensions. FIRE aligns multiple quality signals into a unified space, and integrates diverse data quality raters to provide a comprehensive quality signal for each data point. Further, we introduce a progressive data selection scheme based on FIRE that iteratively refines the selection of high-quality data points. Extensive experiments show that FIRE outperforms other data selection methods and significantly boosts pretrained model performance across a wide range of downstream tasks, while requiring less than 37.5\% of the training data needed by the Random baseline to reach the target performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_00761 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FIRE: Flexible Integration of Data Quality Ratings for Effective Pre-Training Xu, Liangyu Zhang, Xuemiao Duan, Feiyu Wang, Sirui Weng, Rongxiang Wang, Jingang Cai, Xunliang Computation and Language Selecting high-quality data can improve the pretraining efficiency of large language models (LLMs). Existing methods generally rely on heuristic techniques or single quality signals, limiting their ability to evaluate data quality comprehensively. In this work, we propose FIRE, a flexible and scalable framework for integrating multiple data quality raters, which allows for a comprehensive assessment of data quality across various dimensions. FIRE aligns multiple quality signals into a unified space, and integrates diverse data quality raters to provide a comprehensive quality signal for each data point. Further, we introduce a progressive data selection scheme based on FIRE that iteratively refines the selection of high-quality data points. Extensive experiments show that FIRE outperforms other data selection methods and significantly boosts pretrained model performance across a wide range of downstream tasks, while requiring less than 37.5\% of the training data needed by the Random baseline to reach the target performance. |
| title | FIRE: Flexible Integration of Data Quality Ratings for Effective Pre-Training |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.00761 |