Cover Image

Q-Sparse: All Large Language Models can be Fully Sparsely-Activated

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Wang, Hongyu, Ma, Shuming, Wang, Ruiping, Wei, Furu
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

BitNet a4.8: 4-bit Activations for 1-bit LLMs
by: Wang, Hongyu, et al.
Published: (2024)

BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs
by: Wang, Hongyu, et al.
Published: (2025)

The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
by: Ma, Shuming, et al.
Published: (2024)

SparseEval: Efficient Evaluation of Large Language Models by Sparse Optimization
by: Zhang, Taolin, et al.
Published: (2026)

Auto-ICL: In-Context Learning without Human Supervision
by: Yang, Jinghan, et al.
Published: (2023)

CoreInfer: Accelerating Large Language Model Inference with Semantics-Inspired Adaptive Sparse Activation
by: Wang, Qinsi, et al.
Published: (2024)

BitNet b1.58 2B4T Technical Report
by: Ma, Shuming, et al.
Published: (2025)

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
by: Luo, Yuqi, et al.
Published: (2024)

Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Models
by: Deng, Boyi, et al.
Published: (2026)

Sparse Shift Autoencoders for Identifying Concepts from Large Language Model Activations
by: Joshi, Shruti, et al.
Published: (2025)

ROSE: Reordered SparseGPT for More Accurate One-Shot Large Language Models Pruning
by: Su, Mingluo, et al.
Published: (2026)

First Activations Matter: Training-Free Methods for Dynamic Activation in Large Language Models
by: Ma, Chi, et al.
Published: (2024)

EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices
by: Yi, Rongjie, et al.
Published: (2023)

Learn from the Past: Fast Sparse Indexing for Large Language Model Decoding
by: Yao, Feiyu, et al.
Published: (2025)

metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models
by: Kipnis, Alex, et al.
Published: (2024)

Sparse-Autoencoder-Guided Internal Representation Unlearning for Large Language Models
by: Yamashita, Tomoya, et al.
Published: (2025)

Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters
by: Song, Yixin, et al.
Published: (2024)

Textual Aesthetics in Large Language Models
by: Jiang, Lingjie, et al.
Published: (2024)

Scaling Sparse Fine-Tuning to Large Language Models
by: Ansell, Alan, et al.
Published: (2024)

Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantization
by: Son, Seungwoo, et al.
Published: (2024)

In-context Autoencoder for Context Compression in a Large Language Model
by: Ge, Tao, et al.
Published: (2023)

Enhancing One-shot Pruned Pre-trained Language Models through Sparse-Dense-Sparse Mechanism
by: Li, Guanchen, et al.
Published: (2024)

Saten: Sparse Augmented Tensor Networks for Post-Training Compression of Large Language Models
by: Solgi, Ryan, et al.
Published: (2025)

ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models
by: Song, Chenyang, et al.
Published: (2024)

Sparse Layers are Critical to Scaling Looped Language Models
by: Lee, Ryan, et al.
Published: (2026)

Bitnet.cpp: Efficient Edge Inference for Ternary LLMs
by: Wang, Jinheng, et al.
Published: (2025)

NeuroPrune: A Neuro-inspired Topological Sparse Training Algorithm for Large Language Models
by: Dhurandhar, Amit, et al.
Published: (2024)

Model Unlearning via Sparse Autoencoder Subspace Guided Projections
by: Wang, Xu, et al.
Published: (2025)

Sparse is Enough in Fine-tuning Pre-trained Large Language Models
by: Song, Weixi, et al.
Published: (2023)

JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models
by: Dragomir, Alexandra, et al.
Published: (2026)

A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
by: Shu, Dong, et al.
Published: (2025)

Scaling Linear Attention with Sparse State Expansion
by: Pan, Yuqi, et al.
Published: (2025)

Sparse Autoencoders Enable Scalable and Reliable Circuit Identification in Language Models
by: O'Neill, Charles, et al.
Published: (2024)

MediSwift: Efficient Sparse Pre-trained Biomedical Language Models
by: Thangarasa, Vithursan, et al.
Published: (2024)

SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability
by: Karvonen, Adam, et al.
Published: (2025)

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
by: Farnik, Lucy, et al.
Published: (2025)

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
by: Yuan, Jingyang, et al.
Published: (2025)

Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models
by: Wang, Zihan, et al.
Published: (2024)

PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attention
by: Chen, Lida, et al.
Published: (2025)

Steering LLMs? Actually, Sparse Autoencoders can outperform simple baselines
by: Jørgensen, Mikkel Godsk, et al.
Published: (2026)