Cover Image

SparseOpt: Addressing Normalization-induced Gradient Skew in Sparse Training

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Adnan, Mohammed, Jain, Rohan, Jacobs, Tom, Sharma, Ekansh, Krishnan, Rahul G., Burkholz, Rebekka, Ioannou, Yani
Format:	Preprint
Published:	2026
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry
by: Adnan, Mohammed, et al.
Published: (2025)

HORST: Composing Optimizer Geometries for Sparse Transformer Training
by: Jacobs, Tom, et al.
Published: (2026)

Sign-In to the Lottery: Reparameterizing Sparse Training From Scratch
by: Gadhikar, Advait, et al.
Published: (2025)

Mask in the Mirror: Implicit Sparsification
by: Jacobs, Tom, et al.
Published: (2024)

Cyclic Sparse Training: Is it Enough?
by: Gadhikar, Advait, et al.
Published: (2024)

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
by: Jacobs, Tom, et al.
Published: (2025)

Never Saddle for Reparameterized Steepest Descent as Mirror Flow
by: Jacobs, Tom, et al.
Published: (2026)

Dynamic Sparse Training with Structured Sparsity
by: Lasby, Mike, et al.
Published: (2023)

Pay Attention to Small Weights
by: Zhou, Chao, et al.
Published: (2025)

Hyperbolic Aware Minimization: Implicit Bias for Sparsity
by: Jacobs, Tom, et al.
Published: (2025)

Robustness of Mixtures of Experts to Feature Noise
by: Sun, Dong, et al.
Published: (2026)

Learning Fine-grained Parameter Sharing via Sparse Tensor Decomposition
by: Üyük, Cem, et al.
Published: (2024)

GATE: How to Keep Out Intrusive Neighbors
by: Mustafa, Nimrah, et al.
Published: (2024)

Masks, Signs, And Learning Rate Rewinding
by: Gadhikar, Advait, et al.
Published: (2024)

The Graphon Limit Hypothesis: Understanding Neural Network Pruning via Infinite Width Analysis
by: Pham, Hoang, et al.
Published: (2025)

Fixed Aggregation Features Can Rival GNNs
by: Rubio-Madrigal, Celia, et al.
Published: (2026)

What is Left After Distillation? How Knowledge Transfer Impacts Fairness and Bias
by: Mohammadshahi, Aida, et al.
Published: (2024)

Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learning
by: Jacobs, Tom, et al.
Published: (2026)

Spectral Graph Pruning Against Over-Squashing and Over-Smoothing
by: Jamadandi, Adarsh, et al.
Published: (2024)

GNNs Getting ComFy: Community and Feature Similarity Guided Rewiring
by: Rubio-Madrigal, Celia, et al.
Published: (2025)

Multi-Agent Systems are Mixtures of Experts: Who Becomes an Influencer?
by: Bause, Franka, et al.
Published: (2026)

Pruning neural network models for gene regulatory dynamics using data and domain knowledge
by: Hossain, Intekhab, et al.
Published: (2024)

Gradient-Congruity Guided Federated Sparse Training
by: Tian, Chris Xing, et al.
Published: (2024)

When Shift Happens - Confounding Is to Blame
by: Reddy, Abbavaram Gowtham, et al.
Published: (2025)

Frequency-Based Hyperparameter Selection in Games
by: Sanyal, Aniket, et al.
Published: (2026)

SD$^2$: Self-Distilled Sparse Drafters
by: Lasby, Mike, et al.
Published: (2025)

Towards Credit-Fraud Detection via Sparsely Varying Gaussian Approximations
by: Sharma, Harshit, et al.
Published: (2020)

Dynamic Sparse Training of Diagonally Sparse Networks
by: Tyagi, Abhishek, et al.
Published: (2025)

Meta-GCN: A Dynamically Weighted Loss Minimization Method for Dealing with the Data Imbalance in Graph Neural Networks
by: Mohammadizadeh, Mahdi, et al.
Published: (2024)

Sparse-to-Sparse Training of Diffusion Models
by: Oliveira, Inês Cardoso, et al.
Published: (2025)

Sparse-ProxSkip: Accelerated Sparse-to-Sparse Training in Federated Learning
by: Meinhardt, Georg, et al.
Published: (2024)

Order-based Structure Learning with Normalizing Flows
by: Kamkari, Hamidreza, et al.
Published: (2023)

Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients
by: Muhamed, Aashiq, et al.
Published: (2024)

Bridging Domains through Subspace-Aware Model Merging
by: Chaves, Levy, et al.
Published: (2026)

Dense Backpropagation Improves Training for Sparse Mixture-of-Experts
by: Panda, Ashwinee, et al.
Published: (2025)

Transformers with Sparse Attention for Granger Causality
by: Mahesh, Riya, et al.
Published: (2024)

Unlocking the Address Book: Dissecting the Sparse Semantic Structure of LLM Key-Value Caches via Sparse Autoencoders
by: Ma, Qingsen, et al.
Published: (2025)

Differentially Private Optimization with Sparse Gradients
by: Ghazi, Badih, et al.
Published: (2024)

Sparse NMF with Archetypal Regularization: Computational and Robustness Properties
by: Behdin, Kayhan, et al.
Published: (2021)

Navigating Extremes: Dynamic Sparsity in Large Output Spaces
by: Ullah, Nasib, et al.
Published: (2024)