Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2603.00031 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912931478765568 |
|---|---|
| author | Wang, Changhao Yang, Jiaolong Yao, Xinhao Yu, Yunfei Jiao, Peng Yu, Lu Fang, Junpeng Cantoro, Riccardo Cui, Qing Zhou, Jun |
| author_facet | Wang, Changhao Yang, Jiaolong Yao, Xinhao Yu, Yunfei Jiao, Peng Yu, Lu Fang, Junpeng Cantoro, Riccardo Cui, Qing Zhou, Jun |
| contents | The performance of Large Language Models (LLMs) is increasingly governed by data efficiency rather than raw scaling volume. However, existing selection methods often decouple global distribution balancing from local instance selection, compromising the hierarchical integrity of the training set. We introduce \textbf{GRIP} (Geometric Refinement and Adaptive Information Potential), a framework that unifies these dimensions by modeling the corpus as an information-dense geometric space. GRIP employs a \textbf{Rapid Adaptation Probe (RAP)} to quantify the information potential of semantic clusters, dynamically re-allocating the sampling budget to regions with the highest representation deficits. Subsequently, we perform Intra-Cluster Selection using a \textbf{length-rectified geometric prior} to counteract embedding density artifacts and preserve long-tail logical sequences. Extensive evaluations on Mixture-of-Experts (MoE) models up to 300B tokens demonstrate that GRIP consistently outperforms state-of-the-art baselines, \textbf{surpassing the performance of models trained on $3\times$ larger uncurated datasets}. Our work establishes a robust geometric foundation for adaptive data curation in large-scale pre-training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_00031 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | GRIP: Geometric Refinement and Adaptive Information Potential for Data Efficiency Wang, Changhao Yang, Jiaolong Yao, Xinhao Yu, Yunfei Jiao, Peng Yu, Lu Fang, Junpeng Cantoro, Riccardo Cui, Qing Zhou, Jun Computation and Language Machine Learning The performance of Large Language Models (LLMs) is increasingly governed by data efficiency rather than raw scaling volume. However, existing selection methods often decouple global distribution balancing from local instance selection, compromising the hierarchical integrity of the training set. We introduce \textbf{GRIP} (Geometric Refinement and Adaptive Information Potential), a framework that unifies these dimensions by modeling the corpus as an information-dense geometric space. GRIP employs a \textbf{Rapid Adaptation Probe (RAP)} to quantify the information potential of semantic clusters, dynamically re-allocating the sampling budget to regions with the highest representation deficits. Subsequently, we perform Intra-Cluster Selection using a \textbf{length-rectified geometric prior} to counteract embedding density artifacts and preserve long-tail logical sequences. Extensive evaluations on Mixture-of-Experts (MoE) models up to 300B tokens demonstrate that GRIP consistently outperforms state-of-the-art baselines, \textbf{surpassing the performance of models trained on $3\times$ larger uncurated datasets}. Our work establishes a robust geometric foundation for adaptive data curation in large-scale pre-training. |
| title | GRIP: Geometric Refinement and Adaptive Information Potential for Data Efficiency |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2603.00031 |