MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908866306899968 |
|---|---|
| author | Chen, Zhixun Guo, Ping Han, Wenhan Zhang, Yifan Liu, Binbin Lin, Haobin Liu, Fengze Zhao, Yan Zhang, Bingni Wang, Taifeng Zheng, Yin Cohn, Trevor Fang, Meng |
| author_facet | Chen, Zhixun Guo, Ping Han, Wenhan Zhang, Yifan Liu, Binbin Lin, Haobin Liu, Fengze Zhao, Yan Zhang, Bingni Wang, Taifeng Zheng, Yin Cohn, Trevor Fang, Meng |
| contents | Data quality is a critical driver of large language model performance, yet existing model-based selection methods focus almost exclusively on English. We introduce MuRating, a scalable framework that transfers high-quality English data-quality signals into a single rater for 17 target languages. MuRating aggregates multiple English "raters" via pairwise comparisons to learn unified document-quality scores,then projects these judgments through translation to train a multilingual evaluator on monolingual, cross-lingual, and parallel text pairs. Applied to web data, MuRating selects balanced subsets of English and multilingual content to pretrain a 1.2 B-parameter LLaMA model. Compared to strong baselines, including QuRater, AskLLM, DCLM and so on, our approach boosts average accuracy on both English benchmarks and multilingual evaluations, with especially large gains on knowledge-intensive tasks. We further analyze translation fidelity, selection biases, and underrepresentation of narrative material, outlining directions for future work. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_01785 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining Chen, Zhixun Guo, Ping Han, Wenhan Zhang, Yifan Liu, Binbin Lin, Haobin Liu, Fengze Zhao, Yan Zhang, Bingni Wang, Taifeng Zheng, Yin Cohn, Trevor Fang, Meng Computation and Language Artificial Intelligence Machine Learning Data quality is a critical driver of large language model performance, yet existing model-based selection methods focus almost exclusively on English. We introduce MuRating, a scalable framework that transfers high-quality English data-quality signals into a single rater for 17 target languages. MuRating aggregates multiple English "raters" via pairwise comparisons to learn unified document-quality scores,then projects these judgments through translation to train a multilingual evaluator on monolingual, cross-lingual, and parallel text pairs. Applied to web data, MuRating selects balanced subsets of English and multilingual content to pretrain a 1.2 B-parameter LLaMA model. Compared to strong baselines, including QuRater, AskLLM, DCLM and so on, our approach boosts average accuracy on both English benchmarks and multilingual evaluations, with especially large gains on knowledge-intensive tasks. We further analyze translation fidelity, selection biases, and underrepresentation of narrative material, outlining directions for future work. |
| title | MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2507.01785 |