Image de couverture de livre

Loki: Low-rank Keys for Efficient Sparse Attention

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Singhania, Prajwal, Singh, Siddharth, He, Shwai, Feizi, Soheil, Bhatele, Abhinav
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Power Law Guided Dynamic Sifting for Efficient Attention
par: Koley, Nirav, et autres
Publié: (2025)

A 4D Hybrid Algorithm to Scale Parallel Training to Thousands of GPUs
par: Singh, Siddharth, et autres
Publié: (2023)

Speculating Experts Accelerates Inference for Mixture-of-Experts
par: Madan, Vivan, et autres
Publié: (2026)

Understanding and Improving Communication Performance in Multi-node LLM Inference
par: Singhania, Prajwal, et autres
Publié: (2025)

HPC-Coder-V2: Studying Code LLMs Across Low-Resource Parallel Languages
par: Chaturvedi, Aman, et autres
Publié: (2024)

Plexus: Taming Billion-edge Graphs with 3D Parallel Full-graph GNN Training
par: Ranjan, Aditya K., et autres
Publié: (2025)

Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputers
par: Singh, Siddharth, et autres
Publié: (2025)

The Big Send-off: Scalable and Performant Collectives for Deep Learning
par: Singh, Siddharth, et autres
Publié: (2025)

Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning
par: Chegini, Atoosa, et autres
Publié: (2026)

Optimizing Agentic Language Model Inference via Speculative Tool Calls
par: Nichols, Daniel, et autres
Publié: (2025)

Analytics of Longitudinal System Monitoring Data for Performance Prediction
par: Costello, Ian J., et autres
Publié: (2020)

Almost AI, Almost Human: The Challenge of Detecting AI-Polished Writing
par: Saha, Shoumik, et autres
Publié: (2025)

How Learnable Grids Recover Fine Detail in Low Dimensions: A Neural Tangent Kernel Analysis of Multigrid Parametric Encodings
par: Audia, Samuel, et autres
Publié: (2025)

What Matters in Transformers? Not All Attention is Needed
par: He, Shwai, et autres
Publié: (2024)

Chain-of-Defensive-Thought: Structured Reasoning Elicits Robustness in Large Language Models against Reference Corruption
par: Wang, Wenxiao, et autres
Publié: (2025)

Revisiting the Past: Data Unlearning with Model State History
par: Rezaei, Keivan, et autres
Publié: (2025)

Fourier Low-rank and Sparse Tensor for Efficient Tensor Completion
par: Li, Jingyang, et autres
Publié: (2025)

Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques
par: He, Shwai, et autres
Publié: (2024)

Gemstones: A Model Suite for Multi-Faceted Scaling Laws
par: McLeish, Sean, et autres
Publié: (2025)

Maestro: Joint Graph & Config Optimization for Reliable AI Agents
par: Wang, Wenxiao, et autres
Publié: (2025)

RePanda: Pandas-powered Tabular Verification and Reasoning
par: Chegini, Atoosa Malemir, et autres
Publié: (2025)

Decomposing and Interpreting Image Representations via Text in ViTs Beyond CLIP
par: Balasubramanian, Sriram, et autres
Publié: (2024)

Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
par: Geiping, Jonas, et autres
Publié: (2025)

Low-Rank Key Value Attention
par: O'Neill, James, et autres
Publié: (2026)

FLARE: Fast Low-rank Attention Routing Engine
par: Puri, Vedant, et autres
Publié: (2025)

Rethinking Pruning for Vision-Language Models: Strategies for Effective Sparsity and Performance Restoration
par: He, Shwai, et autres
Publié: (2024)

DualSparse-MoE: Coordinating Tensor/Neuron-Level Sparsity with Expert Partition and Reconstruction
par: Cai, Weilin, et autres
Publié: (2025)

QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models
par: Wang, Yutong, et autres
Publié: (2025)

Information Consistent Pruning: How to Efficiently Search for Sparse Networks?
par: Gharatappeh, Soheil, et autres
Publié: (2025)

LOST: Low-rank and Sparse Pre-training for Large Language Models
par: Li, Jiaxi, et autres
Publié: (2025)

Flow-Based Single-Step Completion for Efficient and Expressive Policy Learning
par: Koirala, Prajwal, et autres
Publié: (2025)

Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition
par: He, Zhengfu, et autres
Publié: (2025)

What do we learn from inverting CLIP models?
par: Kazemi, Hamid, et autres
Publié: (2024)

Early Stopping for Large Reasoning Models via Confidence Dynamics
par: Hosseini, Parsa, et autres
Publié: (2026)

Schoenfeld's Anatomy of Mathematical Reasoning by Language Models
par: Li, Ming, et autres
Publié: (2025)

MoEless: Efficient MoE LLM Serving via Serverless Computing
par: Yu, Hanfei, et autres
Publié: (2026)

LOTFormer: Doubly-Stochastic Linear Attention via Low-Rank Optimal Transport
par: Shahbazi, Ashkan, et autres
Publié: (2025)

Low-rank Momentum Factorization for Memory Efficient Training
par: Mahdavinia, Pouria, et autres
Publié: (2025)

SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling
par: Ji, Xiaodong, et autres
Publié: (2025)

CRAUM-Net: Contextual Recursive Attention with Uncertainty Modeling for Salient Object Detection
par: Sagar, Abhinav
Publié: (2020)