Image de couverture de livre

Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixtures

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Nguyen, Dang, Yang, Wenhan, Anand, Rathul, Yang, Yu, Mirzasoleiman, Baharan
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning Artificial Intelligence Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models
par: Yang, Yu, et autres
Publié: (2024)

Better Safe than Sorry: Pre-training CLIP against Targeted Data Poisoning and Backdoor Attacks
par: Yang, Wenhan, et autres
Publié: (2023)

Changing the Training Data Distribution to Reduce Simplicity Bias Improves In-distribution Generalization
par: Nguyen, Dang, et autres
Publié: (2024)

Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity
par: Nguyen, Dang, et autres
Publié: (2025)

Theoretical Perspectives on Data Quality and Synergistic Effects in Pre- and Post-Training Reasoning Models
par: Javanmard, Adel, et autres
Publié: (2026)

Tuning the Implicit Regularizer of Masked Diffusion Language Models: Enhancing Generalization via Insights from $k$-Parity
par: Huang, Jianhao, et autres
Publié: (2026)

Understanding the Role of Training Data in Test-Time Scaling
par: Javanmard, Adel, et autres
Publié: (2025)

Synthetic Text Generation for Training Large Language Models via Gradient Matching
par: Nguyen, Dang, et autres
Publié: (2025)

Data-Efficient Contrastive Self-supervised Learning: Most Beneficial Examples for Supervised Learning Contribute the Least
par: Joshi, Siddharth, et autres
Publié: (2023)

MERIT: Maximum-normalized Element-wise Ratio for Language Model Large-batch Training
par: Luo, Yang, et autres
Publié: (2025)

Graph Contrastive Learning under Heterophily via Graph Filters
par: Yang, Wenhan, et autres
Publié: (2023)

Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
par: Microsoft, et autres
Publié: (2025)

Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictions
par: Xue, Yihao, et autres
Publié: (2025)

MOM: Memory-Efficient Offloaded Mini-Sequence Inference for Long Context Language Models
par: Zhang, Junyang, et autres
Publié: (2025)

Memories Retrieved from Many Paths: A Multi-Prefix Framework for Robust Detection of Training Data Leakage in Large Language Models
par: Dang, Trung Cuong, et autres
Publié: (2025)

Regurgitative Training: The Value of Real Data in Training Large Language Models
par: Zhang, Jinghui, et autres
Publié: (2024)

Few-shot Adaptation to Distribution Shifts By Mixing Source and Target Embeddings
par: Xue, Yihao, et autres
Publié: (2023)

Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Models
par: Zhang, Jun, et autres
Publié: (2025)

MoM: Linear Sequence Modeling with Mixture-of-Memories
par: Du, Jusen, et autres
Publié: (2025)

Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
par: Pan, Bowen, et autres
Publié: (2024)

Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoning
par: Yue, Murong, et autres
Publié: (2023)

TAGCOS: Task-agnostic Gradient Clustered Coreset Selection for Instruction Tuning Data
par: Zhang, Jipeng, et autres
Publié: (2024)

Memory-efficient Energy-adaptive Inference of Pre-Trained Models on Batteryless Embedded Systems
par: Farina, Pietro, et autres
Publié: (2024)

LLMSurgeon: Diagnosing Data Mixture of Large Language Models
par: Luo, Yaxin, et autres
Publié: (2026)

Revisiting Multilingual Data Mixtures in Language Model Pretraining
par: Foroutan, Negar, et autres
Publié: (2025)

Do We Need All the Synthetic Data? Targeted Image Augmentation via Diffusion Models
par: Nguyen, Dang, et autres
Publié: (2025)

LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models
par: Nguyen, Nam V., et autres
Publié: (2024)

Verify when Uncertain: Beyond Self-Consistency in Black Box Hallucination Detection
par: Xue, Yihao, et autres
Publié: (2025)

POET-X: Memory-efficient LLM Training by Scaling Orthogonal Transformation
par: Qiu, Zeju, et autres
Publié: (2026)

Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance
par: Ye, Jiasheng, et autres
Publié: (2024)

AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning
par: Yang, Yifan, et autres
Publié: (2024)

Data Distribution as a Lever for Guiding Optimizers Toward Superior Generalization in LLMs
par: Gangavarapu, Tushaar, et autres
Publié: (2026)

UPCORE: Utility-Preserving Coreset Selection for Balanced Unlearning
par: Patil, Vaidehi, et autres
Publié: (2025)

R&B: Domain Regrouping and Data Mixture Balancing for Efficient Foundation Model Training
par: Ge, Albert, et autres
Publié: (2025)

Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models
par: Kim, Gyeongman, et autres
Publié: (2025)

Boundary-Guided Policy Optimization for Memory-efficient RL of Diffusion Large Language Models
par: Lin, Nianyi, et autres
Publié: (2025)

Improving VTE Identification through Language Models from Radiology Reports: A Comparative Study of Mamba, Phi-3 Mini, and BERT
par: Deng, Jamie, et autres
Publié: (2024)

Language Model Memory and Memory Models for Language
par: Badger, Benjamin L.
Publié: (2026)

LoRA-Mini : Adaptation Matrices Decomposition and Selective Training
par: Singh, Ayush, et autres
Publié: (2024)

Mini-Giants: "Small" Language Models and Open Source Win-Win
par: Zhou, Zhengping, et autres
Publié: (2023)