Cover Image

Group-Level Data Selection for Efficient Pretraining

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Yu, Zichun, Peng, Fei, Lei, Jie, Overwijk, Arnold, Yih, Wen-tau, Xiong, Chenyan
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models
by: Yu, Zichun, et al.
Published: (2024)

Generating Pretraining Tokens from Organic Data for Data-Bound Scaling
by: Yu, Zichun, et al.
Published: (2026)

RePro: Training Language Models to Faithfully Recycle the Web for Pretraining
by: Yu, Zichun, et al.
Published: (2025)

Montessori-Instruct: Generate Influential Training Data Tailored for Student Learning
by: Li, Xiaochuan, et al.
Published: (2024)

FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models
by: Kang, Hao, et al.
Published: (2025)

AutoRule: Reasoning Chain-of-thought Extracted Rule-based Rewards Improve Preference Learning
by: Wang, Tevin, et al.
Published: (2025)

Efficient Multi-Agent System Training with Data Influence-Oriented Tree Search
by: Shi, Wentao, et al.
Published: (2025)

Reliable, Adaptable, and Attributable Language Models with Retrieval
by: Asai, Akari, et al.
Published: (2024)

Few-Shot Data Synthesis for Open Domain Multi-Hop Question Answering
by: Chen, Mingda, et al.
Published: (2023)

ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining
by: Bal, Melis Ilayda, et al.
Published: (2025)

Improving Faithfulness of Abstractive Summarization by Controlling Confounding Effect of Irrelevant Sentences
by: Ghoshal, Asish, et al.
Published: (2022)

Craw4LLM: Efficient Web Crawling for LLM Pretraining
by: Yu, Shi, et al.
Published: (2025)

Enhancing Multilingual LLM Pretraining with Model-Based Data Selection
by: Messmer, Bettina, et al.
Published: (2025)

Analyzing Similarity Metrics for Data Selection for Language Model Pretraining
by: Sam, Dylan, et al.
Published: (2025)

Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length
by: Ma, Xuezhe, et al.
Published: (2024)

Data Mixing for Large Language Models Pretraining: A Survey and Outlook
by: Chen, Zhuo, et al.
Published: (2026)

Stabilizing Efficient Reasoning with Step-Level Advantage Selection
by: Wang, Han, et al.
Published: (2026)

Instruction-tuned Language Models are Better Knowledge Learners
by: Jiang, Zhengbao, et al.
Published: (2024)

SelfCite: Self-Supervised Alignment for Context Attribution in Large Language Models
by: Chuang, Yung-Sung, et al.
Published: (2025)

ED-Copilot: Reduce Emergency Department Wait Time with Language Model Diagnostic Assistance
by: Sun, Liwen, et al.
Published: (2024)

Intercept Cancer: Cancer Pre-Screening with Large Scale Healthcare Foundation Models
by: Sun, Liwen, et al.
Published: (2025)

PithTrain: A Compact and Agent-Native MoE Training System
by: Lai, Ruihang, et al.
Published: (2026)

Efficient Stagewise Pretraining via Progressive Subnetworks
by: Panigrahi, Abhishek, et al.
Published: (2024)

UFO-RL: Uncertainty-Focused Optimization for Efficient Reinforcement Learning Data Selection
by: Zhao, Yang, et al.
Published: (2025)

PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning
by: Wang, Qibin, et al.
Published: (2024)

Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility
by: Szilvasy, Gergely, et al.
Published: (2026)

Improving Pretraining Data Using Perplexity Correlations
by: Thrush, Tristan, et al.
Published: (2024)

Influence Functions for Efficient Data Selection in Reasoning
by: Humane, Prateek, et al.
Published: (2025)

TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection
by: Wu, Wei, et al.
Published: (2024)

DataDecide: How to Predict Best Pretraining Data with Small Experiments
by: Magnusson, Ian, et al.
Published: (2025)

Pruning Literals for Highly Efficient Explainability at Word Level
by: Yadav, Rohan Kumar, et al.
Published: (2024)

Scaling Laws for Mixture Pretraining Under Data Constraints
by: Sedova, Anastasiia, et al.
Published: (2026)

Efficient Data Selection at Scale via Influence Distillation
by: Nikdan, Mahdi, et al.
Published: (2025)

Towards Efficient Active Learning in NLP via Pretrained Representations
by: Vysogorets, Artem, et al.
Published: (2024)

In-context Pretraining: Language Modeling Beyond Document Boundaries
by: Shi, Weijia, et al.
Published: (2023)

Midtraining Bridges Pretraining and Posttraining Distributions
by: Liu, Emmy, et al.
Published: (2025)

PrismRAG: Boosting RAG Factuality with Distractor Resilience and Strategized Reasoning
by: Kachuee, Mohammad, et al.
Published: (2025)

Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection
by: Bethune, Louis, et al.
Published: (2025)

Procedural Pretraining: Warming Up Language Models with Abstract Data
by: Jiang, Liangze, et al.
Published: (2026)

Embedding And Clustering Your Data Can Improve Contrastive Pretraining
by: Merrick, Luke
Published: (2024)