Cover Image

SlimPajama-DC: Understanding Data Combinations for LLM Training

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Shen, Zhiqiang, Tao, Tianhua, Ma, Liqun, Neiswanger, Willie, Liu, Zhengzhong, Wang, Hongyi, Tan, Bowen, Hestness, Joel, Vassilieva, Natalia, Soboleva, Daria, Xing, Eric
Format:	Preprint
Published:	2023
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Crystal: Illuminating LLM Abilities on Language and Code
by: Tao, Tianhua, et al.
Published: (2024)

Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training
by: Bergsma, Shane, et al.
Published: (2025)

GQA-μP: The maximal parameterization update for grouped query attention
by: Chickering, Kyle R., et al.
Published: (2026)

Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs
by: Bergsma, Shane, et al.
Published: (2025)

LLM360 K2: Building a 65B 360-Open-Source Large Language Model from Scratch
by: Liu, Zhengzhong, et al.
Published: (2025)

RedPajama: an Open Dataset for Training Large Language Models
by: Weber, Maurice, et al.
Published: (2024)

Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning
by: Akgül, Ömer Faruk, et al.
Published: (2026)

Agent-Based Insight into Eco-Choices: Simulating the Fast Fashion Shift
by: Soboleva, Daria, et al.
Published: (2024)

Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs
by: Bergsma, Shane, et al.
Published: (2025)

LLM Unlearning Without an Expert Curated Dataset
by: Zhu, Xiaoyuan, et al.
Published: (2025)

MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models
by: Chitty-Venkata, Krishna Teja, et al.
Published: (2025)

Reducing Hyperparameter Tuning Costs in ML, Vision and Language Model Training Pipelines via Memoization-Awareness
by: Essofi, Abdelmajid, et al.
Published: (2024)

From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centered
by: Devic, Siddartha, et al.
Published: (2025)

K2-V2: A 360-Open, Reasoning-Enhanced LLM
by: K2 Team, et al.
Published: (2025)

Scaling with Collapse: Efficient and Predictable Training of LLM Families
by: Bergsma, Shane, et al.
Published: (2025)

DeLLMa: Decision Making Under Uncertainty with Large Language Models
by: Liu, Ollie, et al.
Published: (2024)

Probabilistic Graphical Models: A Concise Tutorial
by: Maasch, Jacqueline, et al.
Published: (2025)

Sparse maximal update parameterization: A holistic approach to sparse training dynamics
by: Dey, Nolan, et al.
Published: (2024)

Supercharging Simulation-Based Inference for Bayesian Optimal Experimental Design
by: Klein, Samuel, et al.
Published: (2026)

Uncertainty Quantification for Forward and Inverse Problems of PDEs via Latent Global Evolution
by: Wu, Tailin, et al.
Published: (2024)

Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions
by: Zhang, Jiarui, et al.
Published: (2024)

Оригинал и карикатура: как британская «профилактическая юстиция» (preventive justice) не приживается на континенте Оторванность от родной почвы выявляет полный абсурд
by: Vassilieva, Elena
Published: (2025)

Создание «уязвимых групп» как механизм обхода правовых гарантий и Прав Человека - Кому выгодно «НЕравенство всех перед судом»?
by: Vassilieva, Elena
Published: (2025)

Creating "Vulnerable Groups" as a Mechanism to Circumvent Legal Guarantees and Human Rights - Who Benefits from "INequality Before the Law"?
by: Vassilieva, Elena
Published: (2025)

Британская модель профилактической юстиции (preventive justice) как новая форма правового подчинения (Claim of Originality)
by: Vassilieva, Elena
Published: (2025)

Claim 5 - The Blind Spot of British Law: Judicial Criminalisation Without Realising It
by: Vassilieva, Elena
Published: (2025)

От «защиты жертв» к системному вредительству: Превентивный надзор как закрепленный в законе обход процессуальных норм
by: Vassilieva, Elena
Published: (2025)

Auditing Black-Box LLM APIs with a Rank-Based Uniformity Test
by: Zhu, Xiaoyuan, et al.
Published: (2025)

SlimDiff: Training-Free, Activation-Guided Hands-free Slimming of Diffusion Models
by: Roy, Arani, et al.
Published: (2025)

RedCoast: A Lightweight Tool to Automate Distributed Training of LLMs on Any GPU/TPUs
by: Tan, Bowen, et al.
Published: (2023)

Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Models
by: Gan, Woody Haosheng, et al.
Published: (2025)

FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation
by: Ma, Liqun, et al.
Published: (2024)

The Balancing Exercise and the Resolution of Rhetorical Antinomies in Judicial Decision‐Making
by: Anita Soboleva
Published: (2024)

Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers
by: Gray, Gavia, et al.
Published: (2024)

Efficient Evaluation of Multi-Task Robot Policies With Active Experiment Selection
by: Anwar, Abrar, et al.
Published: (2025)

LYNX: Learning Dynamic Exits for Confidence-Controlled Reasoning
by: Akgül, Ömer Faruk, et al.
Published: (2025)

Survival Analysis Revisited: Understanding and Unifying Poisson, Exponential, and Cox Models in Fall Risk Analysis
by: Chen, Tianhua
Published: (2025)

Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models
by: Li, Yuan, et al.
Published: (2025)

SlimPipe: Memory-Thrifty and Efficient Pipeline Parallelism for Long-Context LLM Training
by: Li, Zhouyang, et al.
Published: (2025)

AI-University: An LLM-based platform for instructional alignment to scientific classrooms
by: Shojaei, Mostafa Faghih, et al.
Published: (2025)