Imagen de Portada

Distributed Sign Momentum with Local Steps for Training Transformers

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Yu, Shuhua, Zhou, Ding, Xie, Cong, Xu, An, Zhang, Zhi, Liu, Xin, Kar, Soummya
Formato:	Preprint
Publicado:	2024
Materias:	Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Large Deviation Upper Bounds and Improved MSE Rates of Nonlinear SGD: Heavy-tailed Noise and Power of Symmetry
por: Armacki, Aleksandar, et al.
Publicado: (2024)

Nonlinear Stochastic Gradient Descent and Heavy-tailed Noise: A Unified Framework and High-probability Guarantees
por: Armacki, Aleksandar, et al.
Publicado: (2024)

A Unified Framework for Center-based Clustering of Distributed Data
por: Armacki, Aleksandar, et al.
Publicado: (2024)

Distributed Gradient Clustering: Convergence and the Effect of Initialization
por: Armacki, Aleksandar, et al.
Publicado: (2026)

Model-Free Learning and Optimal Policy Design in Multi-Agent MDPs Under Probabilistic Agent Dropout
por: Fiscko, Carmel, et al.
Publicado: (2023)

Distributed gradient methods under heavy-tailed communication noise
por: Vukovic, Manojlo, et al.
Publicado: (2025)

AdaPM: a Partial Momentum Algorithm for LLM Training
por: Zhang, Yimu, et al.
Publicado: (2025)

Smoothed Gradient Clipping and Error Feedback for Decentralized Optimization under Symmetric Heavy-Tailed Noise
por: Yu, Shuhua, et al.
Publicado: (2023)

Decentralized Nonconvex Optimization under Heavy-Tailed Noise: Normalization and Optimal Convergence
por: Yu, Shuhua, et al.
Publicado: (2025)

Sharp High-Probability Rates for Nonlinear SGD under Heavy-Tailed Noise via Symmetrization
por: Armacki, Aleksandar, et al.
Publicado: (2025)

Federated Multi-Objective Learning with Controlled Pareto Frontiers
por: Rao, Jiansheng, et al.
Publicado: (2025)

Improved Analysis for Sign-based Methods with Momentum Updates
por: Jiang, Wei, et al.
Publicado: (2025)

RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimization
por: Deng, Shenyang, et al.
Publicado: (2026)

Tight Long-Term Tail Decay of (Clipped) SGD in Non-Convex Optimization
por: Armacki, Aleksandar, et al.
Publicado: (2026)

Local Steps Speed Up Local GD for Heterogeneous Distributed Logistic Regression
por: Crawshaw, Michael, et al.
Publicado: (2025)

MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router
por: Xie, Yanyue, et al.
Publicado: (2024)

Enhanced Momentum with Momentum Transformers
por: Mason, Max, et al.
Publicado: (2024)

Ordered Local Momentum for Asynchronous Distributed Learning under Arbitrary Delays
por: Shi, Chang-Wei, et al.
Publicado: (2026)

Dynamically Weighted Momentum with Adaptive Step Sizes for Efficient Deep Network Training
por: Wang, Zhifeng, et al.
Publicado: (2025)

High-probability Convergence Bounds for Nonlinear Stochastic Gradient Descent Under Heavy-tailed Noise
por: Armacki, Aleksandar, et al.
Publicado: (2023)

Stochastic Normalized Gradient Descent with Momentum for Large-Batch Training
por: Zhao, Shen-Yi, et al.
Publicado: (2020)

Global Momentum Compression for Sparse Communication in Distributed Learning
por: Shi, Chang-Wei, et al.
Publicado: (2019)

StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models
por: Yu, Dingzhi, et al.
Publicado: (2026)

Self-Explainable Graph Transformer for Link Sign Prediction
por: Li, Lu, et al.
Publicado: (2024)

On the Training Convergence of Transformers for In-Context Classification of Gaussian Mixtures
por: Shen, Wei, et al.
Publicado: (2024)

FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuning
por: Yan, Peishen, et al.
Publicado: (2026)

High-dimensional limit theorems for SGD: Momentum and Adaptive Step-sizes
por: Jagannath, Aukosh, et al.
Publicado: (2025)

Distributed Low-Communication Training with Decoupled Momentum Optimization
por: Nedelkoski, Sasho, et al.
Publicado: (2025)

Beyond the Class Subspace: Teacher-Guided Training for Reliable Out-of-Distribution Detection in Single-Domain Models
por: Yang, Hong, et al.
Publicado: (2026)

PrunedLoRA: Robust Gradient-Based structured pruning for Low-rank Adaptation in Fine-tuning
por: Yu, Xin, et al.
Publicado: (2025)

Enhancing Signed Graph Neural Networks through Curriculum-Based Training
por: Zhang, Zeyu, et al.
Publicado: (2023)

Enhancing Optimizer Stability: Momentum Adaptation of The NGN Step-size
por: Islamov, Rustem, et al.
Publicado: (2025)

Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Order
por: Petrov, Egor, et al.
Publicado: (2025)

On the Optimization and Generalization of Two-layer Transformers with Sign Gradient Descent
por: Li, Bingrui, et al.
Publicado: (2024)

Bridging Training and Merging Through Momentum-Aware Optimization
por: Moayedikia, Alireza, et al.
Publicado: (2025)

Low-rank Momentum Factorization for Memory Efficient Training
por: Mahdavinia, Pouria, et al.
Publicado: (2025)

One-Step Forward and Backtrack: Overcoming Zig-Zagging in Loss-Aware Quantization Training
por: Ma, Lianbo, et al.
Publicado: (2024)

Training-Free ANN-to-SNN Conversion for High-Performance Spiking Transformer
por: Wang, Jingya, et al.
Publicado: (2025)

Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performance
por: Oikonomou, Dimitris, et al.
Publicado: (2024)

Embracing Unknown Step by Step: Towards Reliable Sparse Training in Real World
por: Lei, Bowen, et al.
Publicado: (2024)