Cover Image

GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Chen, Tianhao, Xu, Xin, Liu, Zijing, Li, Pengxiang, Song, Xinyuan, Jaiswal, Ajay Kumar, Zhang, Fan, Hu, Jishan, Wang, Yang, Chen, Hao, Diao, Shizhe, Liu, Shiwei, Li, Yu, Yin, Lu, Yang, Can
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Double-Checker: Enhancing Reasoning of Slow-Thinking LLMs via Self-Critical Fine-Tuning
by: Xu, Xin, et al.
Published: (2025)

Progressive Residual Warmup for Language Model Pretraining
by: Chen, Tianhao, et al.
Published: (2026)

Outlier-weighed Layerwise Sampling for LLM Fine-tuning
by: Li, Pengxiang, et al.
Published: (2024)

Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN
by: Li, Pengxiang, et al.
Published: (2024)

Can We Verify Step by Step for Incorrect Answer Detection?
by: Xu, Xin, et al.
Published: (2024)

FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping
by: Jaiswal, Ajay, et al.
Published: (2024)

The Curse of Depth in Large Language Models
by: Sun, Wenfang, et al.
Published: (2025)

Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients
by: Zhang, Zhenyu, et al.
Published: (2024)

Junk DNA Hypothesis: Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs "Difficult" Downstream Tasks in LLMs
by: Yin, Lu, et al.
Published: (2023)

Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning
by: Bandari, Abhinav, et al.
Published: (2024)

Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding?
by: Li, Pengxiang, et al.
Published: (2026)

UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models
by: Xu, Xin, et al.
Published: (2025)

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
by: Jaiswal, Ajay, et al.
Published: (2024)

Scaled Proximal Gradient Methods for Multiobjective Optimization: Improved Linear Convergence and Nesterov's Acceleration
by: Chen, Jian, et al.
Published: (2024)

ActTail: Global Activation Sparsity in Large Language Models
by: Hou, Wenwen, et al.
Published: (2026)

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
by: Xu, Xin, et al.
Published: (2025)

HELENE: Hessian Layer-wise Clipping and Gradient Annealing for Accelerating Fine-tuning LLM with Zeroth-order Optimization
by: Zhao, Huaqin, et al.
Published: (2024)

Demystifying the Roles of LLM Layers in Retrieval, Knowledge, and Reasoning
by: Song, Xinyuan, et al.
Published: (2025)

Accelerated Gradient Methods with Gradient Restart: Global Linear Convergence
by: Bao, Chenglong, et al.
Published: (2024)

AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMs
by: He, Di, et al.
Published: (2025)

PlanLLM: Video Procedure Planning with Refinable Large Language Models
by: Yang, Dejie, et al.
Published: (2024)

Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding
by: Wu, Chengyue, et al.
Published: (2025)

Variance-Adaptive Muon: Accelerating LLM Pretraining with NSR-Modulated and Variance-Scaled Momentum
by: Li, Jingru, et al.
Published: (2026)

A Study of Student Dependency on Artificial Intelligence Applications in their Education: With Reference to Indore City
by: Ajay Jaiswal
Published: (2025)

Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning
by: Cao, Mingyu, et al.
Published: (2024)

Second-Order Bilevel Optimization with Accelerated Convergence Rates
by: Yang, Sheng, et al.
Published: (2026)

The Global R-linear Convergence of Nesterov's Accelerated Gradient Method with Unknown Strongly Convex Parameter
by: Bao, Chenglong, et al.
Published: (2023)

Fast Convergence of Multiobjective Inertial Gradient Systems with Time Scaling
by: Yin, Yingdong
Published: (2025)

Point Convergence Analysis of the Accelerated Gradient Method for Multiobjective Optimization: Continuous and Discrete
by: Yin, Yingdong
Published: (2025)

Conan-embedding: General Text Embedding with More and Better Negative Samples
by: Li, Shiyu, et al.
Published: (2024)

SMC-AI: Scaling Monte Carlo Simulation to Four Trillion Atoms with AI Accelerators
by: Liu, Xianglin, et al.
Published: (2026)

NCP: Neighborhood-Preserving Non-Uniform Circle Packing for Visualization
by: Li, Duan, et al.
Published: (2026)

Accelerated Gradient Descent for Faster Convergence with Minimal Overhead
by: Graca, Manuel, et al.
Published: (2026)

On the Convergence of Constrained Gradient Method
by: Zhou, Danqing, et al.
Published: (2025)

IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model Pretraining
by: Li, Yixiao, et al.
Published: (2025)

PRESTO: Progressive Pretraining Enhances Synthetic Chemistry Outcomes
by: Cao, He, et al.
Published: (2024)

3D Vision and Language Pretraining with Large-Scale Synthetic Data
by: Yang, Dejie, et al.
Published: (2024)

Accelerated Gradient Methods for Geodesically Convex Optimization: Tractable Algorithms and Convergence Analysis
by: Kim, Jungbin, et al.
Published: (2022)

Towards Computational Microscope of Chemical Order-Disorder via ML-Accelerated Monte Carlo Simulation
by: Zhou, Fanli, et al.
Published: (2026)

Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality
by: Chen, Siyu, et al.
Published: (2024)