Cover Image

Towards the Connection between Activation Sparsity and Flat Minima

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Peng, Ze, Zhang, Jian, Qi, Lei, Gao, Yang, Shi, Yinghuan
Format:	Preprint
Published:	2026
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

On the Implicit Adversariality of Catastrophic Forgetting in Deep Continual Learning
by: Peng, Ze, et al.
Published: (2025)

Are Flat Minima an Illusion?
by: Bennett, Michael Timothy
Published: (2026)

Balanced Direction from Multifarious Choices: Arithmetic Meta-Learning for Domain Generalization
by: Wang, Xiran, et al.
Published: (2025)

Leveraging Flatness to Improve Information-Theoretic Generalization Bounds for SGD
by: Peng, Ze, et al.
Published: (2026)

SAFE: Finding Sparse and Flat Minima to Improve Pruning
by: Lee, Dongyeop, et al.
Published: (2025)

Zeroth-Order Optimization Finds Flat Minima
by: Zhang, Liang, et al.
Published: (2025)

A Flat Minima Perspective on Understanding Augmentations and Model Robustness
by: Yoo, Weebum, et al.
Published: (2025)

A Function-Centric Perspective on Flat and Sharp Minima
by: Mason-Williams, Israel, et al.
Published: (2025)

When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Merging
by: Li, Yayuan, et al.
Published: (2026)

MAGIC: Achieving Superior Model Merging via Magnitude Calibration
by: Li, Yayuan, et al.
Published: (2025)

DP-FedPGN: Finding Global Flat Minima for Differentially Private Federated Learning via Penalizing Gradient Norm
by: Liu, Junkang, et al.
Published: (2025)

WiSparse: Boosting LLM Inference Efficiency with Weight-Aware Mixed Activation Sparsity
by: Chen, Lei, et al.
Published: (2026)

Joint Training Across Multiple Activation Sparsity Regimes
by: Wang, Haotian
Published: (2026)

Post-Training Statistical Calibration for Higher Activation Sparsity
by: Chua, Vui Seng, et al.
Published: (2024)

Activation Sparsity Opportunities for Compressing General Large Language Models
by: Dhar, Nobel, et al.
Published: (2024)

Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Models
by: An, Tai, et al.
Published: (2025)

Uncovering Layer-Dependent Activation Sparsity Patterns in ReLU Transformers
by: Wild, Cody, et al.
Published: (2024)

LibContinual: A Comprehensive Library towards Realistic Continual Learning
by: Li, Wenbin, et al.
Published: (2025)

Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution
by: Park, Jongseok, et al.
Published: (2026)

Beyond Sharp Minima: Robust LLM Unlearning via Feedback-Guided Multi-Point Optimization
by: Wu, Wenhan, et al.
Published: (2025)

Effective Interplay between Sparsity and Quantization: From Theory to Practice
by: Harma, Simla Burcu, et al.
Published: (2024)

Toward Efficient Permutation for Hierarchical N:M Sparsity on GPUs
by: Yu, Seungmin, et al.
Published: (2024)

Accelerating Transformer Inference and Training with 2:4 Activation Sparsity
by: Haziza, Daniel, et al.
Published: (2025)

TriShGAN: Enhancing Sparsity and Robustness in Multivariate Time Series Counterfactuals Explanation
by: Ma, Hongnan, et al.
Published: (2025)

Sparsity-Aware Evolution for Model Merging
by: Zhang, Huan, et al.
Published: (2026)

Improving MLLM Training Efficiency via Stage-Aware Sparsity
by: Shi, Kean, et al.
Published: (2025)

Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes
by: Qiao, Dan, et al.
Published: (2024)

On the Identifiability of Nonlinear ICA: Sparsity and Beyond
by: Zheng, Yujia, et al.
Published: (2022)

BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation
by: Xu, Peng, et al.
Published: (2024)

ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsity
by: Li, Jiaxi, et al.
Published: (2026)

Toward Unifying Group Fairness Evaluation from a Sparsity Perspective
by: Sheng, Zhecheng, et al.
Published: (2025)

Improving Decision Sparsity
by: Sun, Yiyang, et al.
Published: (2024)

Homeostasis and Sparsity in Transformer
by: Kotyuzanskiy, Leonid, et al.
Published: (2024)

Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity
by: Shrestha, Susav, et al.
Published: (2025)

HashAttention: Semantic Sparsity for Faster Inference
by: Desai, Aditya, et al.
Published: (2024)

ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models
by: Song, Chenyang, et al.
Published: (2024)

Towards Robust Influence Functions with Flat Validation Minima
by: Ye, Xichen, et al.
Published: (2025)

LoRA Dropout as a Sparsity Regularizer for Overfitting Control
by: Lin, Yang, et al.
Published: (2024)

Motivating Next-Gen Accelerators with Flexible (N:M) Activation Sparsity via Benchmarking Lightweight Post-Training Sparsification Approaches
by: Alanova, Shirin, et al.
Published: (2025)

Sparsity and Out-of-Distribution Generalization
by: Aaronson, Scott, et al.
Published: (2026)