Image de couverture de livre

SoftSignSGD(S3): An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adam

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Peng, Hanyang, Qin, Shuang, Yu, Yue, Jiang, Fangqing, Wang, Hui, Gao, Wen
Format:	Preprint
Publié:	2025
Sujets:	Machine Learning Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Simple Convergence Proof of Adam From a Sign-like Descent Perspective
par: Peng, Hanyang, et autres
Publié: (2025)

Anon: Extrapolating Adaptivity Beyond SGD and Adam
par: Zhang, Yiheng, et autres
Publié: (2026)

Adaptive Preconditioners Trigger Loss Spikes in Adam
par: Bai, Zhiwei, et autres
Publié: (2025)

StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models
par: Yu, Dingzhi, et autres
Publié: (2026)

Loss Spike in Training Neural Networks
par: Li, Xiaolong, et autres
Publié: (2023)

Sign-SGD via Parameter-Free Optimization
par: Medyakov, Daniil, et autres
Publié: (2025)

Energy-Aware DNN Graph Optimization
par: Wang, Yu, et autres
Publié: (2020)

Comparative Analysis of Novel NIRMAL Optimizer Against Adam and SGD with Momentum
par: Gaud, Nirmal, et autres
Publié: (2025)

Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization
par: Sahu, Sharan, et autres
Publié: (2026)

The Rich and the Simple: On the Implicit Bias of Adam and SGD
par: Vasudeva, Bhavya, et autres
Publié: (2025)

SignSGD with Federated Voting
par: Park, Chanho, et autres
Publié: (2024)

A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD
par: Jin, Ruinan, et autres
Publié: (2024)

The Epochal Sawtooth Phenomenon: Unveiling Training Loss Oscillations in Adam and Other Optimizers
par: Liu, Qi, et autres
Publié: (2024)

Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates
par: Glentis, Athanasios, et autres
Publié: (2026)

Convergence of SGD for Training Neural Networks with Sliced Wasserstein Losses
par: Tanguy, Eloi
Publié: (2023)

Dynamic Regret via Discounted-to-Dynamic Reduction with Applications to Curved Losses and Adam Optimizer
par: Xie, Yan-Feng, et autres
Publié: (2026)

SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training
par: Huang, Tianjin, et autres
Publié: (2025)

Scaling Laws of SignSGD in Linear Regression: When Does It Outperform SGD?
par: Kim, Jihwan, et autres
Publié: (2026)

Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs
par: Mukherjee, Sagnik, et autres
Publié: (2026)

APOLLO: SGD-like Memory, AdamW-level Performance
par: Zhu, Hanqing, et autres
Publié: (2024)

EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models
par: Cheng, Jialiang, et autres
Publié: (2024)

Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Order
par: Petrov, Egor, et autres
Publié: (2025)

Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
par: Wang, Qi, et autres
Publié: (2025)

When and Why SignSGD Outperforms SGD: A Theoretical Study Based on $\ell_1$-norm Lower Bounds
par: Tao, Hongyi, et autres
Publié: (2026)

Structured and Fast Optimization: The Kronecker SGD Algorithm
par: Song, Zhao, et autres
Publié: (2023)

SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Training
par: Sadrtdinov, Ildus, et autres
Publié: (2025)

Enhancing SignSGD: Small-Batch Convergence Analysis and a Hybrid Switching Strategy
par: Chen, Haoran, et autres
Publié: (2026)

Stochastic-Sign SGD for Federated Learning with Theoretical Guarantees
par: Jin, Richeng, et autres
Publié: (2020)

Phases of Muon: When Muon Eclipses SignSGD
par: Paquette, Elliot, et autres
Publié: (2026)

NeMo: A Neuron-Level Modularizing-While-Training Approach for Decomposing DNN Models
par: Bi, Xiaohan, et autres
Publié: (2025)

Genetic and Environmental Risk Factors for Intermittent Explosive Disorder, ADHD and Conduct Disorder: Shared and Unique Influences
par: Fangqing Liu
Publié: (2025)

Frequency-Weighted Training Losses for Phoneme-Level DNN-based Speech Enhancement
par: Monir, Nasser-Eddine, et autres
Publié: (2025)

Collaborative Pareto Set Learning in Multiple Multi-Objective Optimization Problems
par: Shang, Chikai, et autres
Publié: (2024)

SignSGD with Federated Defense: Harnessing Adversarial Attacks through Gradient Sign Decoding
par: Park, Chanho, et autres
Publié: (2024)

Towards Understanding and Enhancing Security of Proof-of-Training for DNN Model Ownership Verification
par: Chang, Yijia, et autres
Publié: (2024)

Cumulative Learning Rate Adaptation: Revisiting Path-Based Schedules for SGD and Adam
par: Atamna, Asma, et autres
Publié: (2025)

Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling
par: Srećković, Teodora, et autres
Publié: (2025)

One-Spike SNN: Single-Spike Phase Coding with Base Manipulation for ANN-to-SNN Conversion Loss Minimization
par: Hwang, Sangwoo, et autres
Publié: (2024)

Adam or Gauss-Newton? A Comparative Study In Terms of Basis Alignment and SGD Noise
par: Liu, Bingbin, et autres
Publié: (2025)

Why Adam Can Beat SGD: Second-Moment Normalization Yields Sharper Tails
par: Jin, Ruinan, et autres
Publié: (2026)