Image de couverture de livre

Batch-Max: Higher LLM Throughput using Larger Batch Sizes and KV Cache Compression

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Metel, Michael R., Chen, Boxing, Rezagholizadeh, Mehdi
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Draft on the Fly: Adaptive Self-Speculative Decoding using Cosine Similarity
par: Metel, Michael R., et autres
Publié: (2024)

On the importance of Data Scale in Pretraining Arabic Language Models
par: Ghaddar, Abbas, et autres
Publié: (2024)

BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batching
par: Zheng, Zhen, et autres
Publié: (2024)

Thinking Long, but Short: Stable Sequential Test-Time Scaling for Large Reasoning Models
par: Metel, Michael R., et autres
Publié: (2026)

X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compression
par: Li, Guihong, et autres
Publié: (2025)

ReGLA: Refining Gated Linear Attention
par: Lu, Peng, et autres
Publié: (2025)

ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
par: Sun, Hanshi, et autres
Publié: (2024)

Multi-Bin Batching for Increasing LLM Inference Throughput
par: Guldogan, Ozgur, et autres
Publié: (2024)

EvolKV: Evolutionary KV Cache Compression for LLM Inference
par: Yu, Bohan, et autres
Publié: (2025)

SABlock: Semantic-Aware KV Cache Eviction with Adaptive Compression Block Size
par: Chen, Jinhan, et autres
Publié: (2025)

Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucination
par: Huang, Jerry, et autres
Publié: (2024)

OTTAWA: Optimal TransporT Adaptive Word Aligner for Hallucination and Omission Translation Errors Detection
par: Huang, Chenyang, et autres
Publié: (2024)

BatchGEMBA: Token-Efficient Machine Translation Evaluation with Batched Prompting and Prompt Compression
par: Larionov, Daniil, et autres
Publié: (2025)

ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
par: Liu, Xiang, et autres
Publié: (2025)

CHARP: Conversation History AwaReness Probing for Knowledge-grounded Dialogue Systems
par: Ghaddar, Abbas, et autres
Publié: (2024)

Sorted LLaMA: Unlocking the Potential of Intermediate Layers of Large Language Models for Dynamic Inference
par: Kavehzadeh, Parsa, et autres
Publié: (2023)

KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head
par: Rehg, Isaac
Publié: (2024)

Are Optimal Algorithms Still Optimal? Rethinking Sorting in LLM-Based Pairwise Ranking with Batching and Caching
par: Wisznia, Juan, et autres
Publié: (2025)

Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity
par: Ma, Da, et autres
Publié: (2024)

NestedKV: Nested Memory Routing for Long-Context KV Cache Compression
par: Chen, Hong, et autres
Publié: (2026)

CaliDrop: KV Cache Compression with Calibration
par: Su, Yi, et autres
Publié: (2025)

PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference
par: Yang, Dongjie, et autres
Publié: (2024)

Lossless KV Cache Compression to 2%
par: Yang, Zhen, et autres
Publié: (2024)

KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMs
par: Chen, Jian, et autres
Publié: (2026)

KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference
par: Tian, Yuxuan, et autres
Publié: (2025)

CodeComp: Structural KV Cache Compression for Agentic Coding
par: Chen, Qiujiang, et autres
Publié: (2026)

EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction
par: Ji, Shiyu, et autres
Publié: (2026)

RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
par: Behnam, Payman, et autres
Publié: (2025)

SemantiCache: Efficient KV Cache Compression via Semantic Chunking and Clustered Merging
par: Wu, Shunlong, et autres
Publié: (2026)

KVSculpt: KV Cache Compression as Distillation
par: Jiang, Bo, et autres
Publié: (2026)

R-KV: Redundancy-aware KV Cache Compression for Reasoning Models
par: Cai, Zefan, et autres
Publié: (2025)

DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs
par: Zhou, Xiabin, et autres
Publié: (2024)

One Size Does Not Fit All: Token-Wise Adaptive Compression for KV Cache
par: Lu, Liming, et autres
Publié: (2026)

PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference
par: Patel, Ishan, et autres
Publié: (2026)

AttentionPredictor: Temporal Patterns Matter for KV Cache Compression
par: Yang, Qingyue, et autres
Publié: (2025)

SmolTulu: Higher Learning Rate to Batch Size Ratios Can Lead to Better Reasoning in SLMs
par: Alrashed, Sultan
Publié: (2024)

S2D: Sorted Speculative Decoding For More Efficient Deployment of Nested Large Language Models
par: Kavehzadeh, Parsa, et autres
Publié: (2024)

Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques
par: Javidnia, Neusha, et autres
Publié: (2025)

HeteroCache: A Dynamic Retrieval Approach to Heterogeneous KV Cache Compression for Long-Context LLM Inference
par: Shi, Zhiyuan, et autres
Publié: (2026)

PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
par: Cai, Zefan, et autres
Publié: (2024)