Copertina

R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Zhang, Zhenyu, Liu, Zechun, Tian, Yuandong, Khaitan, Harshit, Wang, Zhangyang, Li, Steven
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
di: Zhao, Jiawei, et al.
Pubblicazione: (2024)

CLAA: Cross-Layer Attention Aggregation for Accelerating LLM Prefill
di: McDanel, Bradley, et al.
Pubblicazione: (2026)

JacQuant: STE-Free Quantization-Aware Training via Learned Jacobian Surrogates
di: Yi, Kai, et al.
Pubblicazione: (2026)

Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients
di: Zhang, Zhenyu, et al.
Pubblicazione: (2024)

WiSparse: Boosting LLM Inference Efficiency with Weight-Aware Mixed Activation Sparsity
di: Chen, Lei, et al.
Pubblicazione: (2026)

MoE-Spec: Expert Budgeting for Efficient Speculative Decoding
di: McDanel, Bradley, et al.
Pubblicazione: (2026)

WinQ: Accelerating Quantization-Aware Training of Language Models Around Saddle Points
di: Li, Dongyue, et al.
Pubblicazione: (2026)

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
di: Jaiswal, Ajay, et al.
Pubblicazione: (2024)

Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference
di: Dong, Harry, et al.
Pubblicazione: (2024)

Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
di: Tang, Jiaming, et al.
Pubblicazione: (2024)

LoCoCo: Dropping In Convolutions for Long Context Compression
di: Cai, Ruisi, et al.
Pubblicazione: (2024)

Param$Δ$ for Direct Weight Mixing: Post-Train Large Language Model at Zero Cost
di: Cao, Sheng, et al.
Pubblicazione: (2025)

ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsity
di: Li, Jiaxi, et al.
Pubblicazione: (2026)

SparseDVFS: Sparse-Aware DVFS for Energy-Efficient Edge Inference
di: Zhang, Ziyang, et al.
Pubblicazione: (2026)

RaaS: Reasoning-Aware Attention Sparsity for Efficient LLM Reasoning
di: Hu, Junhao, et al.
Pubblicazione: (2025)

Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsity
di: Zhao, Jinze, et al.
Pubblicazione: (2024)

Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity
di: Yin, Lu, et al.
Pubblicazione: (2023)

Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM
di: Solgi, Ryan, et al.
Pubblicazione: (2025)

Neural Networks with Sparse Activation Induced by Large Bias: Tighter Analysis with Bias-Generalized NTK
di: Yang, Hongru, et al.
Pubblicazione: (2023)

Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution
di: Park, Jongseok, et al.
Pubblicazione: (2026)

Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models
di: Zhang, Longteng, et al.
Pubblicazione: (2026)

Activity Sparsity Complements Weight Sparsity for Efficient RNN Inference
di: Mukherji, Rishav, et al.
Pubblicazione: (2023)

Pivoting Factorization: A Compact Meta Low-Rank Representation of Sparsity for Efficient Inference in Large Language Models
di: Zhao, Jialin, et al.
Pubblicazione: (2025)

Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference
di: Liu, Baihui, et al.
Pubblicazione: (2026)

Sparse Attention Remapping with Clustering for Efficient LLM Decoding on PIM
di: Fan, Zehao, et al.
Pubblicazione: (2025)

SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity
di: Khaki, Samir, et al.
Pubblicazione: (2025)

The Path Not Taken: RLVR Provably Learns Off the Principals
di: Zhu, Hanqing, et al.
Pubblicazione: (2025)

GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projection
di: Su, DiJia, et al.
Pubblicazione: (2025)

Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokking
di: Tian, Yuandong
Pubblicazione: (2025)

Quantized Reasoning Models Think They Need to Think Longer, but They Do Not
di: Lotfi, Sanae, et al.
Pubblicazione: (2026)

SVD Contextual Sparsity Predictors for Fast LLM Inference
di: Serbin, Georgii, et al.
Pubblicazione: (2026)

STEM: Scaling Transformers with Embedding Modules
di: Sadhukhan, Ranajoy, et al.
Pubblicazione: (2026)

Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity
di: Shrestha, Susav, et al.
Pubblicazione: (2025)

Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study
di: Zhao, Jinze, et al.
Pubblicazione: (2024)

SpinQuant: LLM quantization with learned rotations
di: Liu, Zechun, et al.
Pubblicazione: (2024)

STS: Efficient Sparse Attention with Speculative Token Sparsity
di: Xu, Ceyu, et al.
Pubblicazione: (2026)

Modality-Aware Zero-Shot Pruning and Sparse Attention for Efficient Multimodal Edge Inference
di: Sui, Yueyuan, et al.
Pubblicazione: (2026)

TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly
di: Koike-Akino, Toshiaki, et al.
Pubblicazione: (2026)

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
di: Luo, Yuqi, et al.
Pubblicazione: (2024)

Reinforcement Learning With Sparse-Executing Actions via Sparsity Regularization
di: Pang, Jing-Cheng, et al.
Pubblicazione: (2021)