Image de couverture de livre

Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMs

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Yang, Jaewoo, Kim, Hayun, Kim, Younghoon
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns
par: Zhao, Ziyu, et autres
Publié: (2026)

Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization
par: Nrusimha, Aniruddha, et autres
Publié: (2024)

Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantization
par: Son, Seungwoo, et autres
Publié: (2024)

LQER: Low-Rank Quantization Error Reconstruction for LLMs
par: Zhang, Cheng, et autres
Publié: (2024)

GLU Attention Improve Transformer
par: Wang, Zehao
Publié: (2025)

GLUScope: A Tool for Analyzing GLU Neurons in Transformer Language Models
par: Gerstner, Sebastian, et autres
Publié: (2026)

Dual Path Attribution: Efficient Attribution for SwiGLU-Transformers through Layer-Wise Target Propagation
par: Jantsch, Lasse Marten, et autres
Publié: (2026)

ZClip: Adaptive Spike Mitigation for LLM Pre-Training
par: Kumar, Abhay, et autres
Publié: (2025)

Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge
par: Shen, Xuan, et autres
Publié: (2023)

Dependency-Aware Semi-Structured Sparsity of GLU Variants in Large Language Models
par: Guo, Zhiyu, et autres
Publié: (2024)

Enhancing Delta Compression in LLMs via SVD-based Quantization Error Minimization
par: Xiong, Boya, et autres
Publié: (2025)

Can LLMs Deceive CLIP? Benchmarking Adversarial Compositionality of Pre-trained Multimodal Representation via Text Updates
par: Ahn, Jaewoo, et autres
Publié: (2025)

L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Models
par: Jeon, Hyesung, et autres
Publié: (2024)

RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantization
par: Huang, Xijie, et autres
Publié: (2024)

Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokens
par: Ouyang, Xu, et autres
Publié: (2024)

Fast Matrix Multiplications for Lookup Table-Quantized LLMs
par: Guo, Han, et autres
Publié: (2024)

SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks
par: Song, Jiwon, et autres
Publié: (2024)

Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?
par: Kim, Jeonghye, et autres
Publié: (2026)

Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Models
par: Park, Jungwoo, et autres
Publié: (2025)

Learning to Correct for QA Reasoning with Black-box LLMs
par: Kim, Jaehyung, et autres
Publié: (2024)

QEFT: Quantization for Efficient Fine-Tuning of LLMs
par: Lee, Changhun, et autres
Publié: (2024)

How Does Quantization Affect Multilingual LLMs?
par: Marchisio, Kelly, et autres
Publié: (2024)

Depth Registers Unlock W4A4 on SwiGLU: A Reader/Generator Decomposition
par: Liu, Ziyang
Publié: (2026)

On the Importance of a Multi-Scale Calibration for Quantization
par: Son, Seungwoo, et autres
Publié: (2026)

QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference
par: Kim, Taesu, et autres
Publié: (2024)

Accurate LoRA-Finetuning Quantization of LLMs via Information Retention
par: Qin, Haotong, et autres
Publié: (2024)

Mitigating Bias in RAG: Controlling the Embedder
par: Kim, Taeyoun, et autres
Publié: (2025)

Interpreting and Mitigating Unwanted Uncertainty in LLMs
par: Roy, Tiasa Singha, et autres
Publié: (2025)

GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
par: Deng, Jianing, et autres
Publié: (2026)

Turning LLM Activations Quantization-Friendly
par: Czakó, Patrik, et autres
Publié: (2025)

LittleBit: Ultra Low-Bit Quantization via Latent Factorization
par: Lee, Banseok, et autres
Publié: (2025)

Few-shot Personalization of LLMs with Mis-aligned Responses
par: Kim, Jaehyung, et autres
Publié: (2024)

A More Accurate Approximation of Activation Function with Few Spikes Neurons
par: Jeong, Dayena, et autres
Publié: (2024)

Rethinking State Tracking in Recurrent Models Through Error Control Dynamics
par: Chung, Jiwan, et autres
Publié: (2026)

CRVQ: Channel-Relaxed Vector Quantization for Extreme Compression of LLMs
par: Xu, Yuzhuang, et autres
Publié: (2024)

LATMiX: Learnable Affine Transformations for Microscaling Quantization of LLMs
par: Gordon, Ofir, et autres
Publié: (2026)

QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resources
par: Li, Zhikai, et autres
Publié: (2023)

AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization
par: IslamBouli, Beshr, et autres
Publié: (2026)

Interpreting the Effects of Quantization on LLMs
par: Singh, Manpreet, et autres
Publié: (2025)

A Concise Review of Hallucinations in LLMs and their Mitigation
par: Pulkundwar, Parth, et autres
Publié: (2025)