Cover Image

Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Song, Yixin, Xie, Haotong, Zhang, Zhengyan, Wen, Bo, Ma, Li, Mi, Zeyu, Chen, Haibo
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU
by: Song, Yixin, et al.
Published: (2023)

QiMeng-Attention: SOTA Attention Operator is generated by SOTA Attention Algorithm
by: Zhou, Qirui, et al.
Published: (2025)

ReLU$^2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs
by: Zhang, Zhengyan, et al.
Published: (2024)

Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning
by: Yuan, Yurun, et al.
Published: (2025)

ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models
by: Song, Chenyang, et al.
Published: (2024)

QUAD: Quantization and Parameter-Efficient Tuning of LLM with Activation Decomposition
by: Hu, Yuxuan, et al.
Published: (2025)

Q-Sparse: All Large Language Models can be Fully Sparsely-Activated
by: Wang, Hongyu, et al.
Published: (2024)

PowerInfer-2: Fast Large Language Model Inference on a Smartphone
by: Xue, Zhenliang, et al.
Published: (2024)

Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning
by: Liu, Yong, et al.
Published: (2024)

SVFit: Parameter-Efficient Fine-Tuning of Large Pre-Trained Models Using Singular Values
by: Sun, Chengwei, et al.
Published: (2024)

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
by: Yuan, Jingyang, et al.
Published: (2025)

DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
by: Song, Chenyang, et al.
Published: (2026)

CoreInfer: Accelerating Large Language Model Inference with Semantics-Inspired Adaptive Sparse Activation
by: Wang, Qinsi, et al.
Published: (2024)

DropLoRA: Sparse Low-Rank Adaptation for Parameter-Efficient Fine-Tuning
by: Zhang, Haojie
Published: (2025)

Exploring Activation Patterns of Parameters in Language Models
by: Wang, Yudong, et al.
Published: (2024)

Query Performance Explanation through Large Language Model for HTAP Systems
by: Xiu, Haibo, et al.
Published: (2024)

HSR-Enhanced Sparse Attention Acceleration
by: Chen, Bo, et al.
Published: (2024)

GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling
by: Chen, Tianhao, et al.
Published: (2025)

From Sparse Dependence to Sparse Attention: Unveiling How Chain-of-Thought Enhances Transformer Sample Efficiency
by: Wen, Kaiyue, et al.
Published: (2024)

Dense vs Sparse Pretraining at Tiny Scale: Active-Parameter vs Total-Parameter Matching
by: Wael, Abdalrahman
Published: (2026)

SAEMark: Steering Personalized Multilingual LLM Watermarks with Sparse Autoencoders
by: Yu, Zhuohao, et al.
Published: (2025)

From Parameters to Data: A Task-Parameter-Guided Fine-Tuning Pipeline for Efficient LLM Alignment
by: Chen, Hao, et al.
Published: (2026)

What Matters in LLM-generated Data: Diversity and Its Effect on Model Fine-Tuning
by: Zhu, Yuchang, et al.
Published: (2025)

SqueezeLLM: Dense-and-Sparse Quantization
by: Kim, Sehoon, et al.
Published: (2023)

Edit Less, Achieve More: Dynamic Sparse Neuron Masking for Lifelong Knowledge Editing in LLMs
by: Liu, Jinzhe, et al.
Published: (2025)

Context-Alignment: Activating and Enhancing LLM Capabilities in Time Series
by: Hu, Yuxiao, et al.
Published: (2025)

Internal Activation Revision: Safeguarding Vision Language Models Without Parameter Update
by: Li, Qing, et al.
Published: (2025)

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
by: Luo, Yuqi, et al.
Published: (2024)

Parameter Importance is Not Static: Evolving Parameter Isolation for Supervised Fine-Tuning
by: Lin, Zekai, et al.
Published: (2026)

SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression
by: Wang, Xin, et al.
Published: (2024)

Sparse is Enough in Fine-tuning Pre-trained Large Language Models
by: Song, Weixi, et al.
Published: (2023)

SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models
by: Huang, Wei, et al.
Published: (2024)

Learn to Memorize: Optimizing LLM-based Agents with Adaptive Memory Framework
by: Zhang, Zeyu, et al.
Published: (2025)

Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training
by: Liu, Yixin, et al.
Published: (2026)

AMAQ: Adaptive Mixed-bit Activation Quantization for Collaborative Parameter Efficient Fine-tuning
by: Song, Yurun, et al.
Published: (2025)

Supervised Fine-Tuning Achieve Rapid Task Adaption Via Alternating Attention Head Activation Patterns
by: Zhao, Yang, et al.
Published: (2024)

Merging Beyond: Streaming LLM Updates via Activation-Guided Rotations
by: Yao, Yuxuan, et al.
Published: (2026)

GaLLoP: Gradient-based Sparse Learning on Low-Magnitude Parameters
by: Choudhary, Anand, et al.
Published: (2025)

InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
by: Xiao, Chaojun, et al.
Published: (2024)

LLM Router: Rethinking Routing with Prefill Activations
by: Varshney, Tanay, et al.
Published: (2026)