Image de couverture de livre

Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Yin, Lu, Wu, You, Zhang, Zhenyu, Hsieh, Cheng-Yu, Wang, Yaqing, Jia, Yiling, Li, Gen, Jaiswal, Ajay, Pechenizkiy, Mykola, Liang, Yi, Bendersky, Michael, Wang, Zhangyang, Liu, Shiwei
Format:	Preprint
Publié:	2023
Sujets:	Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Leave it to the Specialist: Repair Sparse LLMs with Sparse Fine-Tuning via Sparsity Evolution
par: Xiao, Qiao, et autres
Publié: (2025)

Junk DNA Hypothesis: Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs "Difficult" Downstream Tasks in LLMs
par: Yin, Lu, et autres
Publié: (2023)

Maximum Redundancy Pruning: A Principle-Driven Layerwise Sparsity Allocation for LLMs
par: Gao, Chang, et autres
Publié: (2025)

Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity
par: Liu, Shiwei, et autres
Publié: (2021)

Adaptive Sparsity Level during Training for Efficient Time Series Forecasting with Transformers
par: Atashgahi, Zahra, et autres
Publié: (2023)

OWLed: Outlier-weighed Layerwise Pruning for Efficient Autonomous Driving Framework
par: Li, Jiaxi, et autres
Publié: (2024)

Outlier-weighed Layerwise Sampling for LLM Fine-tuning
par: Li, Pengxiang, et autres
Publié: (2024)

(PASS) Visual Prompt Locates Good Structure Sparsity through a Recurrent HyperNetwork
par: Huang, Tianjin, et autres
Publié: (2024)

Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients
par: Zhang, Zhenyu, et autres
Publié: (2024)

Boosting Robustness in Preference-Based Reinforcement Learning with Dynamic Sparsity
par: Muslimani, Calarina, et autres
Publié: (2024)

R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference
par: Zhang, Zhenyu, et autres
Publié: (2025)

Addressing the Collaboration Dilemma in Low-Data Federated Learning via Transient Sparsity
par: Xiao, Qiao, et autres
Publié: (2025)

Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning
par: Cao, Mingyu, et autres
Publié: (2024)

Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning
par: Bandari, Abhinav, et autres
Publié: (2024)

Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization
par: Li, Guanchen, et autres
Publié: (2025)

DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMs
par: Yin, Ruokai, et autres
Publié: (2025)

Towards Extreme Pruning of LLMs with Plug-and-Play Mixed Sparsity
par: Xu, Chi, et autres
Publié: (2025)

Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling
par: Xiao, Qiao, et autres
Publié: (2026)

When Does Sparsity Mitigate the Curse of Depth in LLMs
par: Muhtar, Dilxat, et autres
Publié: (2026)

One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs
par: He, Di, et autres
Publié: (2026)

Pushing the Limits of Sparsity: A Bag of Tricks for Extreme Pruning
par: Li, Andy, et autres
Publié: (2024)

Twilight: Adaptive Attention Sparsity with Hierarchical Top-$p$ Pruning
par: Lin, Chaofan, et autres
Publié: (2025)

In Search of Adam's Secret Sauce
par: Orvieto, Antonio, et autres
Publié: (2025)

ActTail: Global Activation Sparsity in Large Language Models
par: Hou, Wenwen, et autres
Publié: (2026)

Compressing LLMs: The Truth is Rarely Pure and Never Simple
par: Jaiswal, Ajay, et autres
Publié: (2023)

Dynamic Data Pruning for Automatic Speech Recognition
par: Xiao, Qiao, et autres
Publié: (2024)

Scaling Attention via Feature Sparsity
par: Xie, Yan, et autres
Publié: (2026)

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
par: Jaiswal, Ajay, et autres
Publié: (2024)

CRISP: Hybrid Structured Sparsity for Class-aware Model Pruning
par: Aggarwal, Shivam, et autres
Publié: (2023)

Selection Plateau and a Sparsity-Dependent Hierarchy of Pruning Features
par: Li, Guangqi, et autres
Publié: (2026)

Improving Decision Sparsity
par: Sun, Yiyang, et autres
Publié: (2024)

Beyond One-Way Pruning: Bidirectional Pruning-Regrowth for Extreme Accuracy-Sparsity Tradeoff
par: Liu, Junchen, et autres
Publié: (2025)

Sparsity Forcing: Reinforcing Token Sparsity of MLLMs
par: Chen, Feng, et autres
Publié: (2025)

SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruning
par: Liao, Huanxuan, et autres
Publié: (2025)

ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsity
par: Li, Jiaxi, et autres
Publié: (2026)

FedPaI: Achieving Extreme Sparsity in Federated Learning via Pruning at Initialization
par: Wang, Haonan, et autres
Publié: (2025)

Is there "Secret Sauce'' in Large Language Model Development?
par: Mertens, Matthias, et autres
Publié: (2026)

Sirius: Contextual Sparsity with Correction for Efficient LLMs
par: Zhou, Yang, et autres
Publié: (2024)

You Can Have Better Graph Neural Networks by Not Training Weights at All: Finding Untrained GNNs Tickets
par: Huang, Tianjin, et autres
Publié: (2022)

Pruning as Evolution: Emergent Sparsity Through Selection Dynamics in Neural Networks
par: Shah, Zubair, et autres
Publié: (2026)