Cover Image

Progressive Residual Warmup for Language Model Pretraining

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Chen, Tianhao, Xu, Xin, Yin, Lu, Chen, Hao, Wang, Yang, Diao, Shizhe, Yang, Can
Format:	Preprint
Published:	2026
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
by: Xu, Xin, et al.
Published: (2025)

Can We Verify Step by Step for Incorrect Answer Detection?
by: Xu, Xin, et al.
Published: (2024)

GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling
by: Chen, Tianhao, et al.
Published: (2025)

Double-Checker: Enhancing Reasoning of Slow-Thinking LLMs via Self-Critical Fine-Tuning
by: Xu, Xin, et al.
Published: (2025)

Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models
by: Xu, Xin, et al.
Published: (2026)

UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models
by: Xu, Xin, et al.
Published: (2025)

Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners
by: Xu, Xin, et al.
Published: (2025)

Active Prompting with Chain-of-Thought for Large Language Models
by: Diao, Shizhe, et al.
Published: (2023)

Warmup-Distill: Bridge the Distribution Mismatch between Teacher and Student before Knowledge Distillation
by: Sun, Zengkui, et al.
Published: (2025)

Mitigating Hidden Confounding by Progressive Confounder Imputation via Large Language Models
by: Yang, Hao, et al.
Published: (2025)

ConstraintChecker: A Plugin for Large Language Models to Reason on Commonsense Knowledge Bases
by: Do, Quyet V., et al.
Published: (2024)

ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
by: Liu, Mingjie, et al.
Published: (2025)

$\textbf{PLUM}$: Improving Code LMs with Execution-Guided On-Policy Preference Learning Driven By Synthetic Test Cases
by: Zhang, Dylan, et al.
Published: (2024)

FIRST: Teach A Reliable Large Language Model Through Efficient Trustworthy Distillation
by: Shum, KaShun, et al.
Published: (2024)

Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data
by: Shum, KaShun, et al.
Published: (2023)

Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training
by: Diao, Shizhe, et al.
Published: (2025)

ProRank: Prompt Warmup via Reinforcement Learning for Small Language Models Reranking
by: Li, Xianming, et al.
Published: (2025)

R-Tuning: Instructing Large Language Models to Say `I Don't Know'
by: Zhang, Hanning, et al.
Published: (2023)

Conan-embedding: General Text Embedding with More and Better Negative Samples
by: Li, Shiyu, et al.
Published: (2024)

The Warmup Dilemma: How Learning Rate Strategies Impact Speech-to-Text Model Convergence
by: Gaido, Marco, et al.
Published: (2025)

The Role of Mixed-Language Documents for Multilingual Large Language Model Pretraining
by: Shao, Jiandong, et al.
Published: (2026)

Multilingual Language Model Pretraining using Machine-translated Data
by: Wang, Jiayi, et al.
Published: (2025)

Mangosteen: An Open Thai Corpus for Language Model Pretraining
by: Phatthiyaphaibun, Wannaphong, et al.
Published: (2025)

Evaluating Large Language Models for Evidence-Based Clinical Question Answering
by: Wang, Can, et al.
Published: (2025)

Open-SQL Framework: Enhancing Text-to-SQL on Open-source Large Language Models
by: Chen, Xiaojun, et al.
Published: (2024)

PretrainRL: Alleviating Factuality Hallucination of Large Language Models at the Beginning
by: Liu, Langming, et al.
Published: (2026)

ReaLM: Residual Quantization Bridging Knowledge Graph Embeddings and Large Language Models
by: Guo, Wenbin, et al.
Published: (2025)

Attention Residuals
by: Kimi Team, et al.
Published: (2026)

CodeGraph: Enhancing Graph Reasoning of LLMs with Code
by: Cai, Qiaolong, et al.
Published: (2024)

SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales
by: Xu, Tianyang, et al.
Published: (2024)

On the importance of Data Scale in Pretraining Arabic Language Models
by: Ghaddar, Abbas, et al.
Published: (2024)

A Systematic Survey of Semantic Role Labeling in the Era of Pretrained Language Models
by: Chen, Huiyao, et al.
Published: (2025)

Hymba: A Hybrid-head Architecture for Small Language Models
by: Dong, Xin, et al.
Published: (2024)

Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining
by: Zhu, Jinchang, et al.
Published: (2026)

CCI4.0: A Bilingual Pretraining Dataset for Enhancing Reasoning in Large Language Models
by: Liu, Guang, et al.
Published: (2025)

Length Value Model: Scalable Value Pretraining for Token-Level Length Modeling
by: Zhang, Zhen, et al.
Published: (2026)

Adapt-Pruner: Adaptive Structural Pruning for Efficient Small Language Model Training
by: Pan, Rui, et al.
Published: (2025)

Toward Automated Cognitive Assessment in Parkinson's Disease Using Pretrained Language Models
by: Khanna, Varada, et al.
Published: (2025)

Multi-granularity Interactive Attention Framework for Residual Hierarchical Pronunciation Assessment
by: Han, Hong, et al.
Published: (2026)

Progressively Label Enhancement for Large Language Model Alignment
by: Liu, Biao, et al.
Published: (2024)