Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2602.03772 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908809812770816 |
|---|---|
| author | Wang, Changhao Yu, Yunfei Yao, Xinhao Yang, Jiaolong Cantoro, Riccardo Li, Chaobo Cui, Qing Zhou, Jun |
| author_facet | Wang, Changhao Yu, Yunfei Yao, Xinhao Yang, Jiaolong Cantoro, Riccardo Li, Chaobo Cui, Qing Zhou, Jun |
| contents | The scaling of Large Language Models (LLMs) is increasingly limited by data quality. Most methods handle data mixing and sample selection separately, which can break the structure in code corpora. We introduce \textbf{UniGeM}, a framework that unifies mixing and selection by treating data curation as a \textit{manifold approximation} problem without training proxy models or relying on external reference datasets. UniGeM operates hierarchically: \textbf{Macro-Exploration} learns mixing weights with stability-based clustering; \textbf{Micro-Mining} filters high-quality instances by their geometric distribution to ensure logical consistency. Validated by training 8B and 16B MoE models on 100B tokens, UniGeM achieves \textbf{2.0$\times$ data efficiency} over a random baseline and further improves overall performance compared to SOTA methods in reasoning-heavy evaluations and multilingual generalization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_03772 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | UniGeM: Unifying Data Mixing and Selection via Geometric Exploration and Mining Wang, Changhao Yu, Yunfei Yao, Xinhao Yang, Jiaolong Cantoro, Riccardo Li, Chaobo Cui, Qing Zhou, Jun Machine Learning Artificial Intelligence The scaling of Large Language Models (LLMs) is increasingly limited by data quality. Most methods handle data mixing and sample selection separately, which can break the structure in code corpora. We introduce \textbf{UniGeM}, a framework that unifies mixing and selection by treating data curation as a \textit{manifold approximation} problem without training proxy models or relying on external reference datasets. UniGeM operates hierarchically: \textbf{Macro-Exploration} learns mixing weights with stability-based clustering; \textbf{Micro-Mining} filters high-quality instances by their geometric distribution to ensure logical consistency. Validated by training 8B and 16B MoE models on 100B tokens, UniGeM achieves \textbf{2.0$\times$ data efficiency} over a random baseline and further improves overall performance compared to SOTA methods in reasoning-heavy evaluations and multilingual generalization. |
| title | UniGeM: Unifying Data Mixing and Selection via Geometric Exploration and Mining |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2602.03772 |