Cover Image

SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Mozaffari, Mohammad, Yazdanbakhsh, Amir, Zhang, Zhao, Dehnavi, Maryam Mehri
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

OPTIMA: Optimal One-shot Pruning for LLMs via Quadratic Programming Reconstruction
by: Mozaffari, Mohammad, et al.
Published: (2025)

SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
by: Mozaffari, Mohammad, et al.
Published: (2024)

PATCH: Learnable Tile-level Hybrid Sparsity for LLMs
by: Hourri, Younes, et al.
Published: (2025)

MKOR: Momentum-Enabled Kronecker-Factor-Based Optimizer Using Rank-1 Updates
by: Mozaffari, Mohammad, et al.
Published: (2023)

Hierarchical Sparse Plus Low Rank Compression of LLM
by: Kumar, Pawan, et al.
Published: (2025)

Sparse High Rank Adapters
by: Bhardwaj, Kartikeya, et al.
Published: (2024)

LEAP: Learnable End-to-End Adaptive Pruning of Large Language Models
by: Mozaffari, Mohammad, et al.
Published: (2026)

OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition
by: Zhang, Stephen, et al.
Published: (2024)

Towards Symmetric Low-Rank Adapters
by: Panoutsos, Tales, et al.
Published: (2025)

Ensembles of Low-Rank Expert Adapters
by: Li, Yinghao, et al.
Published: (2025)

ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization
by: You, Haoran, et al.
Published: (2024)

LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference
by: Fu, Qichen, et al.
Published: (2024)

Flora: Low-Rank Adapters Are Secretly Gradient Compressors
by: Hao, Yongchang, et al.
Published: (2024)

Aggregating Low Rank Adapters in Federated Fine-tuning
by: Trautmann, Evelyn, et al.
Published: (2025)

LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformers
by: Shen, Xuan, et al.
Published: (2024)

Beyond Moore's Law: Harnessing the Redshift of Generative AI with Effective Hardware-Software Co-Design
by: Yazdanbakhsh, Amir
Published: (2025)

Training-Free Bayesianization for Low-Rank Adapters of Large Language Models
by: Shi, Haizhou, et al.
Published: (2024)

TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Models
by: Liu, Zuxin, et al.
Published: (2023)

AdapterTune: Zero-Initialized Low-Rank Adapters for Frozen Vision Transformers
by: Khazem, Salim
Published: (2026)

LoRA-Pro: Are Low-Rank Adapters Properly Optimized?
by: Wang, Zhengbo, et al.
Published: (2024)

Multiple Choice Learning of Low-Rank Adapters for Language Modeling
by: Letzelter, Victor, et al.
Published: (2025)

PoLAR: Polar-Decomposed Low-Rank Adapter Representation
by: Lion, Kai, et al.
Published: (2025)

Plasticity vs. Rigidity: The Impact of Low-Rank Adapters on Reasoning on a Micro-Budget
by: Khan, Zohaib, et al.
Published: (2026)

Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMs
by: Ni, Wentao, et al.
Published: (2026)

Low-Rank Adapters Meet Neural Architecture Search for LLM Compression
by: Muñoz, J. Pablo, et al.
Published: (2025)

zFLoRA: Zero-Latency Fused Low-Rank Adapters
by: Gowda, Dhananjaya, et al.
Published: (2025)

LoRe: Personalizing LLMs via Low-Rank Reward Modeling
by: Bose, Avinandan, et al.
Published: (2025)

FL-TAC: Enhanced Fine-Tuning in Federated Learning via Low-Rank, Task-Specific Adapter Clustering
by: Ping, Siqi, et al.
Published: (2024)

UniPruning: Unifying Local Metric and Global Feedback for Scalable Sparse LLMs
by: Ding, Yizhuo, et al.
Published: (2025)

IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model Pretraining
by: Li, Yixiao, et al.
Published: (2025)

Sparse Adapter Fusion for Continual Learning in NLP
by: Zeng, Min, et al.
Published: (2026)

Finer Parameter Steps for Low-Rank PEFT: A Controlled Study with CP Tensor Adapters
by: Wang, Xinjue, et al.
Published: (2026)

RankCLIP: Ranking-Consistent Language-Image Pretraining
by: Zhang, Yiming, et al.
Published: (2024)

Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuning
by: Cong, Peizhuang, et al.
Published: (2025)

PHLoRA: data-free Post-hoc Low-Rank Adapter extraction from full-rank checkpoint
by: Vasani, Bhoomit, et al.
Published: (2025)

Random Initialization of Gated Sparse Adapters
by: Retault, Vi, et al.
Published: (2025)

Training Neural Networks from Scratch with Parallel Low-Rank Adapters
by: Huh, Minyoung, et al.
Published: (2024)

Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications
by: Wei, Boyi, et al.
Published: (2024)

Low-Rank Compression of Pretrained Models via Randomized Subspace Iteration
by: Pourkamali-Anaraki, Farhad
Published: (2026)

BoostLoRA: Growing Effective Rank by Boosting Adapters
by: Anantha, Raviteja, et al.
Published: (2026)