Image de couverture de livre

How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Luo, Kairong, Sun, Zhenbo, Wen, Haodong, Shi, Xinyu, Cui, Jiarui, Dang, Chenyi, Lyu, Kaifeng, Chen, Wenguang
Format:	Preprint
Publié:	2025
Sujets:	Machine Learning Artificial Intelligence Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules
par: Luo, Kairong, et autres
Publié: (2025)

Larger Datasets Can Be Repeated More: A Theoretical Analysis of Multi-Epoch Scaling in Linear Regression
par: Yan, Tingkai, et autres
Publié: (2025)

PCMind-2.1-Kaiyuan-2B Technical Report
par: Luo, Kairong, et autres
Publié: (2025)

Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold
par: Li, Xinghan, et autres
Publié: (2025)

RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval
par: Wen, Kaiyue, et autres
Publié: (2024)

DataDecide: How to Predict Best Pretraining Data with Small Experiments
par: Magnusson, Ian, et autres
Publié: (2025)

False Discovery Rate Control For Structured Multiple Testing: Asymmetric Rules And Conformal Q-values
par: Zhao, Zinan, et autres
Publié: (2023)

LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?
par: Tang, Kexian, et autres
Publié: (2025)

Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining
par: Feng, Steven, et autres
Publié: (2024)

Curriculum Learning for LLM Pretraining: An Analysis of Learning Dynamics
par: Elgaar, Mohamed, et autres
Publié: (2026)

Long-term hydro-ecological regime shifts and river–lake connectivity changes in the Jingjiang reach and Yangtze–Dongting Lake system following Three Gorges Reservoir operation
par: Wenguang, Luo
Publié: (2026)

Efficient Stagewise Pretraining via Progressive Subnetworks
par: Panigrahi, Abhishek, et autres
Publié: (2024)

The Power of Power Law: Asymmetry Enables Compositional Reasoning
par: Wang, Zixuan, et autres
Publié: (2026)

How to Choose the Best Handyman Service app for Your Needs
par: joy, Swiza
Publié: (2026)

Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs
par: Bergsma, Shane, et autres
Publié: (2025)

The Marginal Value of Momentum for Small Learning Rate SGD
par: Wang, Runzhe, et autres
Publié: (2023)

The Finetuner's Fallacy: When to Pretrain with Your Finetuning Data
par: Baek, Christina, et autres
Publié: (2026)

Displaced Fermionic Gaussian States and their Classical Simulation
par: Lyu, Xingjian, et autres
Publié: (2024)

Fermionic Gaussian Testing and Non-Gaussian Measures via Convolution
par: Lyu, Xingjian, et autres
Publié: (2024)

Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Data
par: Zhang, Xuemiao, et autres
Publié: (2025)

Toward General Semantic Chunking: A Discriminative Framework for Ultra-Long Documents
par: Wu, Kaifeng, et autres
Publié: (2025)

Omni-CLST: Error-aware Curriculum Learning with guided Selective chain-of-Thought for audio question answering
par: Zhao, Jinghua, et autres
Publié: (2025)

Electrolyte Design for High–Rate Performance Biphasic Membrane–Free Batteries
par: Xinyu Li, et autres
Publié: (2026)

Embedding And Clustering Your Data Can Improve Contrastive Pretraining
par: Merrick, Luke
Publié: (2024)

Self-Evolving Curriculum for LLM Reasoning
par: Chen, Xiaoyin, et autres
Publié: (2025)

Is Podcasting Your Best Alternative?
Publié: (2024)

Shift is Good: Mismatched Data Mixing Improves Test Performance
par: Medvedev, Marko, et autres
Publié: (2025)

Data Mixing Can Induce Phase Transitions in Knowledge Acquisition
par: Gu, Xinran, et autres
Publié: (2025)

Interplay of Institutional Ownership and Negative Media Coverage on Firms’ High‐Investment Human Resource Management Systems
par: Kaifeng Jiang, et autres
Publié: (2025)

Beyond Repetition: Text Simplification and Curriculum Learning for Data-Constrained Pretraining
par: Roque, Matthew Theodore, et autres
Publié: (2025)

A Study of Decay Rate of Bound Negative Muons
par: Deng, Jian-Bo, et autres
Publié: (2024)

Your Pretrained Model Tells the Difficulty Itself: A Self-Adaptive Curriculum Learning Paradigm for Natural Language Understanding
par: Feng, Qi, et autres
Publié: (2025)

Sparse Recovery With Multiple Data Streams: A Sequential Adaptive Testing Approach
par: Wang, Weinan, et autres
Publié: (2017)

Research on Dynamic Mechanical Behavior of Reinforcing Steel Under Ultrahigh Strain Rate
par: Haodong Sun, et autres
Publié: (2024)

Pick the Best Platform for Your Content
Publié: (2024)

When Bias Pretends to Be Truth: How Spurious Correlations Undermine Hallucination Detection in LLMs
par: Wang, Shaowen, et autres
Publié: (2025)

TIIF-Bench: How Does Your T2I Model Follow Your Instructions?
par: Wei, Xinyu, et autres
Publié: (2025)

Craw4LLM: Efficient Web Crawling for LLM Pretraining
par: Yu, Shi, et autres
Publié: (2025)

DCL-SE: Dynamic Curriculum Learning for Spatiotemporal Encoding of Brain Imaging
par: Zhou, Meihua, et autres
Publié: (2025)

ModelLens: Finding the Best for Your Task from Myriads of Models
par: Cai, Rui, et autres
Publié: (2026)