Image de couverture de livre

Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteur principal:	Kurt, Uygar
Format:	Preprint
Publié:	2026
Sujets:	Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Forensic Implications of Localized AI: Artifact Analysis of Ollama, LM Studio, and llama.cpp
par: Murtuza, Shariq
Publié: (2026)

Pruning vs Quantization: Which is Better?
par: Kuzmin, Andrey, et autres
Publié: (2023)

Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders
par: He, Zhengfu, et autres
Publié: (2024)

Inspection and Control of Self-Generated-Text Recognition Ability in Llama3-8b-Instruct
par: Ackerman, Christopher, et autres
Publié: (2024)

Domain Adaptation of Llama3-70B-Instruct through Continual Pre-Training and Model Merging: A Comprehensive Evaluation
par: Siriwardhana, Shamane, et autres
Publié: (2024)

Llama-3.1-FoundationAI-SecurityLLM-Reasoning-8B Technical Report
par: Yang, Zhuoran, et autres
Publié: (2026)

Production-Grade Local LLM Inference on Apple Silicon: A Comparative Study of MLX, MLC-LLM, Ollama, llama.cpp, and PyTorch MPS
par: Rajesh, Varun, et autres
Publié: (2025)

FP8 Quantization: The Power of the Exponent
par: Kuzmin, Andrey, et autres
Publié: (2022)

Compression Scaling Laws:Unifying Sparsity and Quantization
par: Frantar, Elias, et autres
Publié: (2025)

UniComp: A Unified Evaluation of Large Language Model Compression via Pruning, Quantization and Distillation
par: von Rad, Jonathan, et autres
Publié: (2026)

A Quantized VAE-MLP Botnet Detection Model: A Systematic Evaluation of Quantization-Aware Training and Post-Training Quantization Strategies
par: Wasswa, Hassan, et autres
Publié: (2025)

QuantKAN: A Unified Quantization Framework for Kolmogorov Arnold Networks
par: Fuad, Kazi Ahmed Asif, et autres
Publié: (2025)

Which Augmentation Should I Use? An Empirical Investigation of Augmentations for Self-Supervised Phonocardiogram Representation Learning
par: Ballas, Aristotelis, et autres
Publié: (2023)

SqueezeLLM: Dense-and-Sparse Quantization
par: Kim, Sehoon, et autres
Publié: (2023)

Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysis
par: Zhao, Jiaqi, et autres
Publié: (2025)

Unified Stochastic Framework for Neural Network Quantization and Pruning
par: Zhang, Haoyu, et autres
Publié: (2024)

An Empirical Study of Qwen3 Quantization
par: Zheng, Xingyu, et autres
Publié: (2025)

A Comprehensive Evaluation on Quantization Techniques for Large Language Models
par: Liu, Yutong, et autres
Publié: (2025)

A Systematic Evaluation of On-Device LLMs: Quantization, Performance, and Resources
par: Song, Qingyu, et autres
Publié: (2025)

Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Use
par: Delavande, Julien, et autres
Publié: (2026)

Activation Sensitivity as a Unifying Principle for Post-Training Quantization
par: Xu, Bruce Changlong
Publié: (2026)

QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resources
par: Li, Zhikai, et autres
Publié: (2023)

SKIM: Any-bit Quantization Pushing The Limits of Post-Training Quantization
par: Bai, Runsheng, et autres
Publié: (2024)

Learning Unified User Quantized Tokenizers for User Representation
par: He, Chuan, et autres
Publié: (2025)

Layer-wise Quantization for Quantized Optimistic Dual Averaging
par: Nguyen, Anh Duc, et autres
Publié: (2025)

Operationalizing Quantized Disentanglement
par: Barin-Pacela, Vitoria, et autres
Publié: (2025)

Matryoshka Quantization
par: Nair, Pranav, et autres
Publié: (2025)

How to Parameterize Asymmetric Quantization Ranges for Quantization-Aware Training
par: You, Jaeseong, et autres
Publié: (2024)

KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
par: Hooper, Coleman, et autres
Publié: (2024)

Efficient Evaluation of Quantization-Effects in Neural Codecs
par: Mack, Wolfgang, et autres
Publié: (2025)

LPCD: Unified Framework from Layer-Wise to Submodule Quantization
par: Ichikawa, Yuma, et autres
Publié: (2025)

Online Vector Quantized Attention
par: Alonso, Nick, et autres
Publié: (2026)

On the Spectral Flattening of Quantized Embeddings
par: Huang, Junlin, et autres
Publié: (2026)

Pyramid Vector Quantization for LLMs
par: van der Ouderaa, Tycho F. A., et autres
Publié: (2024)

CodeQuant: Unified Clustering and Quantization for Enhanced Outlier Smoothing in Low-Precision Mixture-of-Experts
par: Yin, Xiangyang, et autres
Publié: (2026)

UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMs
par: Chiang, Hung-Yueh, et autres
Publié: (2025)

Preserve-Then-Quantize: Balancing Rank Budgets for Quantization Error Reconstruction in LLMs
par: Cho, Yoonjun, et autres
Publié: (2026)

Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization
par: Arai, Yamato, et autres
Publié: (2025)

Divide-or-Conquer? Which Part Should You Distill Your LLM?
par: Wu, Zhuofeng, et autres
Publié: (2024)

Efficient Post-training Quantization with FP8 Formats
par: Shen, Haihao, et autres
Publié: (2023)