Cover Image

Projected Compression: Trainable Projection for Efficient Transformer Compression

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Stefaniak, Maciej, Krutul, Michał, Małaśnicki, Jan, Pióro, Maciej, Krajewski, Jakub, Jaszczur, Sebastian, Cygan, Marek, Adamczewski, Kamil, Ludziejewski, Jan
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
by: Ludziejewski, Jan, et al.
Published: (2025)

Decoupled Relative Learning Rate Schedules
by: Ludziejewski, Jan, et al.
Published: (2025)

$μ$-Parametrization for Mixture of Experts
by: Małaśnicki, Jan, et al.
Published: (2025)

MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts
by: Pióro, Maciej, et al.
Published: (2024)

Scaling Laws for Fine-Grained Mixture of Experts
by: Krajewski, Jakub, et al.
Published: (2024)

Mixture of Tokens: Continuous MoE through Cross-Example Aggregation
by: Antoniak, Szymon, et al.
Published: (2023)

MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection
by: Lin, Bokai, et al.
Published: (2024)

One Shot vs. Iterative: Rethinking Pruning Strategies for Model Compression
by: Janusz, Mikołaj, et al.
Published: (2025)

State Soup: In-Context Skill Learning, Retrieval and Mixing
by: Pióro, Maciej, et al.
Published: (2024)

Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
by: Krajewski, Jakub, et al.
Published: (2025)

Efficient LLM Moderation with Multi-Layer Latent Prototypes
by: Chrabąszcz, Maciej, et al.
Published: (2025)

Communication Compression for Tensor Parallel LLM Inference
by: Hansen-Palmus, Jan, et al.
Published: (2024)

Strategic Fusion Optimizes Transformer Compression
by: Rahman, Md Shoaibur
Published: (2025)

EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projection
by: Zhou, Yuhao, et al.
Published: (2025)

Attention Sinks as Internal Signals for Hallucination Detection in Large Language Models
by: Binkowski, Jakub, et al.
Published: (2026)

Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training
by: Krajewski, Jakub, et al.
Published: (2025)

SEE: Sememe Entanglement Encoding for Transformer-bases Models Compression
by: Zhang, Jing, et al.
Published: (2024)

Palu: Compressing KV-Cache with Low-Rank Projection
by: Chang, Chi-Chih, et al.
Published: (2024)

Prune-Quantize-Distill: An Ordered Pipeline for Efficient Neural Network Compression
by: Zhou, Longsheng, et al.
Published: (2026)

ConCISE: Confidence-guided Compression in Step-by-step Efficient Reasoning
by: Qiao, Ziqing, et al.
Published: (2025)

Adapting LLMs for Efficient Context Processing through Soft Prompt Compression
by: Wang, Cangqing, et al.
Published: (2024)

Efficient Prompt Tuning by Multi-Space Projection and Prompt Fusion
by: Lan, Pengxiang, et al.
Published: (2024)

ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization
by: Yadav, Prateek, et al.
Published: (2023)

Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models
by: Muñoz, J. Pablo, et al.
Published: (2025)

Trainable Dynamic Mask Sparse Attention
by: Shi, Jingze, et al.
Published: (2025)

LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Models
by: Li, Guangyan, et al.
Published: (2024)

Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy
by: Li, Pingzhi, et al.
Published: (2023)

KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference
by: Tian, Yuxuan, et al.
Published: (2025)

GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM
by: Kang, Hao, et al.
Published: (2024)

Model Compression and Efficient Inference for Large Language Models: A Survey
by: Wang, Wenxiao, et al.
Published: (2024)

On the Compressibility of Quantized Large Language Models
by: Mao, Yu, et al.
Published: (2024)

KVSculpt: KV Cache Compression as Distillation
by: Jiang, Bo, et al.
Published: (2026)

When Compression Meets Model Compression: Memory-Efficient Double Compression for Large Language Models
by: Wang, Weilan, et al.
Published: (2025)

Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention
by: Lv, Xingtai, et al.
Published: (2024)

TrimR: Verifier-based Training-Free Thinking Compression for Efficient Test-Time Scaling
by: Lin, Weizhe, et al.
Published: (2025)

ROCKET: Rapid Optimization via Calibration-guided Knapsack Enhanced Truncation for Efficient Model Compression
by: Ali, Ammar, et al.
Published: (2026)

Cross-lingual Named Entity Corpus for Slavic Languages
by: Piskorski, Jakub, et al.
Published: (2024)

Fast Vocabulary Transfer for Language Model Compression
by: Gee, Leonidas, et al.
Published: (2024)

Learning to Compress Prompt in Natural Language Formats
by: Chuang, Yu-Neng, et al.
Published: (2024)

LILO: Learning Interpretable Libraries by Compressing and Documenting Code
by: Grand, Gabriel, et al.
Published: (2023)