Image de couverture de livre

Optimizing Pretraining Data Mixtures with LLM-Estimated Utility

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Held, William, Paranjape, Bhargavi, Koura, Punit Singh, Lewis, Mike, Zhang, Frank, Mihaylov, Todor
Format:	Preprint
Publié:	2025
Sujets:	Computation and Language Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Husky: A Unified, Open-Source Language Agent for Multi-Step Reasoning
par: Kim, Joongwon, et autres
Publié: (2024)

Revisiting Multilingual Data Mixtures in Language Model Pretraining
par: Foroutan, Negar, et autres
Publié: (2025)

MixAtlas: Uncertainty-aware Data Mixture Optimization for Multimodal LLM Midtraining
par: Wen, Bingbing, et autres
Publié: (2026)

HorizonBench: Long-Horizon Personalization with Evolving Preferences
par: Li, Shuyue Stella, et autres
Publié: (2026)

Distilling an End-to-End Voice Assistant Without Instruction Training Data
par: Held, William, et autres
Publié: (2024)

Tracing Persona Vectors Through LLM Pretraining
par: Moskvoretskii, Viktor, et autres
Publié: (2026)

daVinci-LLM:Towards the Science of Pretraining
par: Qin, Yiwei, et autres
Publié: (2026)

IGOT: Information Gain Optimized Tokenizer on Domain Adaptive Pretraining
par: Feng, Dawei, et autres
Publié: (2024)

Assessing and Verifying Task Utility in LLM-Powered Applications
par: Arabzadeh, Negar, et autres
Publié: (2024)

Personalized Decision Modeling: Utility Optimization or Textualized-Symbolic Reasoning
par: Zhao, Yibo, et autres
Publié: (2025)

Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Data
par: Zhang, Xuemiao, et autres
Publié: (2025)

Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model
par: Du, Xinrun, et autres
Publié: (2024)

Scaling LLM Inference with Optimized Sample Compute Allocation
par: Zhang, Kexun, et autres
Publié: (2024)

In-context Pretraining: Language Modeling Beyond Document Boundaries
par: Shi, Weijia, et autres
Publié: (2023)

Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining
par: Feng, Steven, et autres
Publié: (2024)

A Continued Pretrained LLM Approach for Automatic Medical Note Generation
par: Yuan, Dong, et autres
Publié: (2024)

Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining
par: Guo, Ping, et autres
Publié: (2025)

On Linear Representations and Pretraining Data Frequency in Language Models
par: Merullo, Jack, et autres
Publié: (2025)

MASS: Mathematical Data Selection via Skill Graphs for Pretraining Large Language Models
par: Li, Jiazheng, et autres
Publié: (2025)

MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applications
par: Kanithi, Praveenkumar, et autres
Publié: (2024)

Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels
par: Cen, Zhepeng, et autres
Publié: (2025)

Estimating LLM Uncertainty with Evidence
par: Ma, Huan, et autres
Publié: (2025)

Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance
par: Ye, Jiasheng, et autres
Publié: (2024)

Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data
par: Wang, Xinyi, et autres
Publié: (2024)

LLM-Specific Utility: A New Perspective for Retrieval-Augmented Generation
par: Zhang, Hengran, et autres
Publié: (2025)

HarmonyGuard: Toward Safety and Utility in Web Agents via Adaptive Policy Enhancement and Dual-Objective Optimization
par: Chen, Yurun, et autres
Publié: (2025)

AI-LieDar: Examine the Trade-off Between Utility and Truthfulness in LLM Agents
par: Su, Zhe, et autres
Publié: (2024)

Towards better Human-Agent Alignment: Assessing Task Utility in LLM-Powered Applications
par: Arabzadeh, Negar, et autres
Publié: (2024)

Collab: Controlled Decoding using Mixture of Agents for LLM Alignment
par: Chakraborty, Souradip, et autres
Publié: (2025)

Utility-Focused LLM Annotation for Retrieval and Retrieval-Augmented Generation
par: Zhang, Hengran, et autres
Publié: (2025)

Output Embedding Centering for Stable LLM Pretraining
par: Stollenwerk, Felix, et autres
Publié: (2026)

Next Token Knowledge Tracing: Exploiting Pretrained LLM Representations to Decode Student Behaviour
par: Norris, Max, et autres
Publié: (2025)

Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment
par: Gan, Woody Haosheng, et autres
Publié: (2026)

FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Model
par: Wu, Xiaobao, et autres
Publié: (2024)

Bias Mitigation Agent: Optimizing Source Selection for Fair and Balanced Knowledge Retrieval
par: Singh, Karanbir, et autres
Publié: (2025)

How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
par: Luo, Kairong, et autres
Publié: (2025)

Beyond Repetition: Text Simplification and Curriculum Learning for Data-Constrained Pretraining
par: Roque, Matthew Theodore, et autres
Publié: (2025)

BhashaKritika: Building Synthetic Pretraining Data at Scale for Indic Languages
par: Manoj, Guduru, et autres
Publié: (2025)

Arctic-SnowCoder: Demystifying High-Quality Data in Code Pretraining
par: Wei, Yuxiang, et autres
Publié: (2024)

Toward Cross-Lingual Quality Classifiers for Multilingual Pretraining Data Selection
par: Turki, Yassine, et autres
Publié: (2026)