Image de couverture de livre

Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Ye, Jiasheng, Liu, Peiju, Sun, Tianxiang, Zhan, Jun, Zhou, Yunhua, Qiu, Xipeng
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language Artificial Intelligence Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?
par: Zeng, Zhiyuan, et autres
Publié: (2025)

Domain2Vec: Vectorizing Datasets to Find the Optimal Data Mixture without Training
par: Zhang, Mozhi, et autres
Publié: (2025)

Explicit Multi-head Attention for Inter-head Interaction in Large Language Models
par: Peng, Runyu, et autres
Publié: (2026)

AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
par: Zhan, Jun, et autres
Publié: (2024)

Turn Waste into Worth: Rectifying Top-$k$ Router of MoE
par: Zeng, Zhiyuan, et autres
Publié: (2024)

BiMix: A Bivariate Data Mixing Law for Language Model Pretraining
par: Ge, Ce, et autres
Publié: (2024)

BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments
par: Wang, Xinghao, et autres
Publié: (2024)

Evolution of Concepts in Language Model Pre-Training
par: Ge, Xuyang, et autres
Publié: (2025)

MixAtlas: Uncertainty-aware Data Mixture Optimization for Multimodal LLM Midtraining
par: Wen, Bingbing, et autres
Publié: (2026)

Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning
par: Ye, Jiasheng, et autres
Publié: (2023)

The Open-World Lottery Ticket Hypothesis for OOD Intent Classification
par: Zhou, Yunhua, et autres
Publié: (2022)

LLMSurgeon: Diagnosing Data Mixture of Large Language Models
par: Luo, Yaxin, et autres
Publié: (2026)

Aioli: A Unified Optimization Framework for Language Model Data Mixing
par: Chen, Mayee F., et autres
Publié: (2024)

Agent Alignment in Evolving Social Norms
par: Li, Shimin, et autres
Publié: (2024)

Observational Scaling Laws and the Predictability of Language Model Performance
par: Ruan, Yangjun, et autres
Publié: (2024)

Rethinking Data Mixing from the Perspective of Large Language Models
par: Xu, Yuanjian, et autres
Publié: (2026)

Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning
par: Zhang, Yechen, et autres
Publié: (2026)

Relative-Based Scaling Law for Neural Language Models
par: Yue, Baoqing, et autres
Publié: (2025)

Revisiting Multilingual Data Mixtures in Language Model Pretraining
par: Foroutan, Negar, et autres
Publié: (2025)

Data-Driven Calibration of Prediction Sets in Large Vision-Language Models Based on Inductive Conformal Prediction
par: Ye, Yuanchang, et autres
Publié: (2025)

RegMix: Data Mixture as Regression for Language Model Pre-training
par: Liu, Qian, et autres
Publié: (2024)

Task-Stratified Knowledge Scaling Laws for Post-Training Quantized Large Language Models
par: Zhou, Chenxi, et autres
Publié: (2025)

Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models -- The Story Goes On
par: Zeng, Liang, et autres
Publié: (2024)

MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Merging
par: Wang, Jiapeng, et autres
Publié: (2026)

Balanced Data Sampling for Language Model Training with Clustering
par: Shao, Yunfan, et autres
Publié: (2024)

Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging
par: Yang, Jinluan, et autres
Publié: (2025)

Linear Model Merging Unlocks Simple and Scalable Multimodal Data Mixture Optimization
par: Berasi, Davide, et autres
Publié: (2026)

Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixtures
par: Nguyen, Dang, et autres
Publié: (2024)

Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections
par: Wang, Bo, et autres
Publié: (2025)

Efficient End-to-end Language Model Fine-tuning on Graphs
par: Xue, Rui, et autres
Publié: (2023)

Towards a Comprehensive Scaling Law of Mixture-of-Experts
par: Zhao, Guoliang, et autres
Publié: (2025)

Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models
par: Guo, Yiran, et autres
Publié: (2025)

Collaborative Performance Prediction for Large Language Models
par: Zhang, Qiyuan, et autres
Publié: (2024)

Scaling Laws for Predicting Downstream Performance in LLMs
par: Chen, Yangyi, et autres
Publié: (2024)

A Law of Next-Token Prediction in Large Language Models
par: He, Hangfeng, et autres
Publié: (2024)

Diversity-oriented Data Augmentation with Large Language Models
par: Wang, Zaitian, et autres
Publié: (2025)

Evaluating the Performance of Large Language Models on GAOKAO Benchmark
par: Zhang, Xiaotian, et autres
Publié: (2023)

Towards the Law of Capacity Gap in Distilling Language Models
par: Zhang, Chen, et autres
Publié: (2023)

Mixture of Heterogeneous Grouped Experts for Language Modeling
par: Ma, Zhicheng, et autres
Publié: (2026)

Beyond Attention Magnitude: Leveraging Inter-layer Rank Consistency for Efficient Vision-Language-Action Models
par: Liu, Peiju, et autres
Publié: (2026)