Cover Image

Enhancing One-shot Pruned Pre-trained Language Models through Sparse-Dense-Sparse Mechanism

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Li, Guanchen, Zhao, Xiandong, Liu, Lian, Li, Zeping, Li, Dong, Tian, Lu, He, Jie, Sirasao, Ashish, Barsoum, Emad
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Sparse Laneformer
by: Liu, Ji, et al.
Published: (2024)

Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization
by: Li, Guanchen, et al.
Published: (2025)

Dual LoRA: Enhancing LoRA with Magnitude and Direction Updates
by: Xu, Yixing, et al.
Published: (2025)

Amphista: Bi-directional Multi-head Decoding for Accelerating LLM Inference
by: Li, Zeping, et al.
Published: (2024)

SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruning
by: Liao, Huanxuan, et al.
Published: (2025)

SparseLLM: Towards Global Pruning for Pre-trained Language Models
by: Bai, Guangji, et al.
Published: (2024)

DL-QAT: Weight-Decomposed Low-Rank Quantization-Aware Training for Large Language Models
by: Ke, Wenjin, et al.
Published: (2025)

DiffSparse: Accelerating Diffusion Transformers with Learned Token Sparsity
by: Zhu, Haowei, et al.
Published: (2026)

MonoGS++: Fast and Accurate Monocular RGB Gaussian SLAM
by: Li, Renwu, et al.
Published: (2025)

Sparse Meets Dense: Unified Generative Recommendations with Cascaded Sparse-Dense Representations
by: Yang, Yuhao, et al.
Published: (2025)

SDFP: Speculative Decoding with FIT-Pruned Models for Training-Free and Plug-and-Play LLM Acceleration
by: Wei, Hanyu, et al.
Published: (2026)

LOST: Low-rank and Sparse Pre-training for Large Language Models
by: Li, Jiaxi, et al.
Published: (2025)

Learnable Permutation for Structured Sparsity on Transformer Models
by: Li, Zekai, et al.
Published: (2026)

LADDER: An Efficient Framework for Video Frame Interpolation
by: Shen, Tong, et al.
Published: (2024)

MSWA: Refining Local Attention with Multi-ScaleWindow Attention
by: Xu, Yixing, et al.
Published: (2025)

Towards Scale-Aware Full Surround Monodepth with Transformers
by: Yang, Yuchen, et al.
Published: (2024)

SparseFocus: Learning-based One-shot Autofocus for Microscopy with Sparse Content
by: Zhai, Yongping, et al.
Published: (2025)

S-STE: Continuous Pruning Function for Efficient 2:4 Sparse Pre-training
by: Hu, Yuezhou, et al.
Published: (2024)

Sparse is Enough in Fine-tuning Pre-trained Large Language Models
by: Song, Weixi, et al.
Published: (2023)

Provable Sparse Inversion and Token Relabel Enhanced One-shot Federated Learning with ViTs
by: Shen, Li, et al.
Published: (2026)

FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing
by: Li, Zekai, et al.
Published: (2024)

The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws
by: Jin, Tian, et al.
Published: (2025)

Recover-LoRA: Data-Free Accuracy Recovery of Degraded Language Models via Low-Rank Adaptation
by: Das, Devleena, et al.
Published: (2025)

Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match
by: Li, Jinze, et al.
Published: (2025)

SqueezeLLM: Dense-and-Sparse Quantization
by: Kim, Sehoon, et al.
Published: (2023)

X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compression
by: Li, Guihong, et al.
Published: (2025)

RadCLIP: Enhancing Radiologic Image Analysis through Contrastive Language-Image Pre-training
by: Lu, Zhixiu, et al.
Published: (2024)

TernaryLLM: Ternarized Large Language Model
by: Chen, Tianqi, et al.
Published: (2024)

DiP-GO: A Diffusion Pruner via Few-step Gradient Optimization
by: Zhu, Haowei, et al.
Published: (2024)

ConDense: Consistent 2D/3D Pre-training for Dense and Sparse Features from Multi-View Images
by: Zhang, Xiaoshuai, et al.
Published: (2024)

The Right to be Forgotten in Pruning: Unveil Machine Unlearning on Sparse Models
by: Xiao, Yang, et al.
Published: (2025)

Forecasting LLM Inference Performance via Hardware-Agnostic Analytical Modeling
by: Patwari, Rajeev, et al.
Published: (2025)

PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptation
by: An, Zihao, et al.
Published: (2025)

AMD-Hummingbird: Towards an Efficient Text-to-Video Model
by: Isobe, Takashi, et al.
Published: (2025)

E-MMDiT: Revisiting Multimodal Diffusion Transformer Design for Fast Image Synthesis under Limited Resources
by: Shen, Tong, et al.
Published: (2025)

Athena: Enhancing Multimodal Reasoning with Data-efficient Process Reward Models
by: Wang, Shuai, et al.
Published: (2025)

Beyond the Target: From Imitation to Collaboration in Speculative Decoding
by: Li, Jinze, et al.
Published: (2026)

DOC-GS: Dual-Domain Observation and Calibration for Reliable Sparse-View Gaussian Splatting
by: Li, Hantang, et al.
Published: (2026)

MediSwift: Efficient Sparse Pre-trained Biomedical Language Models
by: Thangarasa, Vithursan, et al.
Published: (2024)

Sparse Reasoning is Enough: Biological-Inspired Framework for Video Anomaly Detection with Large Pre-trained Models
by: Huang, He, et al.
Published: (2025)